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Our Conference and the Performance 
Assessment Committee ^ 



Skip Kifer 



As I think most are aware there is a rather 
short time frame for implementing perfor- 
mance assessments in Kentucky schools. 
The legislation calls for an interim state-wide assess- 
ment in 1991- 1992 and a " full-blown" assessment 
in 1993 - 1994. In addition, there are requirements 
for implementing simultaneously a program of con- 
tinuous assessment in each of the Commonwealth's 
schools. 

The legislation also sets a procedure for designing 
and implementing the assessment program. Among 
the procedures are: a committee of assessment 
experts is to write a Request for Proposals (RFP) 
that contains a framework for the assessment; a 
contract is to be awarded to a successful bidder who 
will carry out the assessments, and; the successful 
bidder wiU work with schools to produce the results 
of the assessment. 

I am a member of the Committee that is to write 
the RFP and that is what I would like to talk about 
briefly . I hasten to say that I can speak about the 
committee and its work but I cannot speak for the 
committee. Other members of that committee in- 
clude: Pat Forgione who presides over and con- 
ducts in Connecticut what is considered to be an 
exemplary state assessment program; Jason 
Millman, from Cornell university, whose expertise 
is n the technical arena and experience includes 
having served on the technical committee for the 
California state assessment; Doris Redfield who is 
an evaluator from the Center for the Study of 
Evaluation at UCLA and who has worked closely in 
the recent past with Kentucky schools; Grant Wig- 
gins who is the well-known proponent of perfor- 
mance based assessment. 

Members of the committee are familiar with major 
assessment programs in the United States (Califor- 
nia. Connecticut, New York, Vermont) as weU as 



those which are well-known internationally (e.g.. 
Great Britain and Australia). They are, I believe,* 
sensitive to both the potential for these kinds of 
assessments and the problems they produce. 

The Committee was appointed in late November 
and has met tv/ice to discuss the fiamework for the 
assessment and the structure of the RFP. There are 
plans to meet again later this month. 

The Committee has spent time understanding the 
nature of the legislation and the requirements of the 
Kentucky assessments. It has benefited from dis- 
cussions with those who framed the law and these 
who are working on other aspects of the reform 
which have implications for performance testing. 
For example, we talked with Roger Pankratz, the 
leader of the task forces working on framing goals 
and defining measurements for the assessment, on 
both occasions that we met. In addition, Roger has 
made sure that there has been a continuous flow of 
mformation from bis groups to our committee. 

Because the Kentucky legislation has components 
smiilar to those in other states, neither the committee 
nor the Commonwealth will be starting at point zero 
in terms of our assessment. Existing practices in 
other states and countries should provide a basis for 
framing the state-wide performance assessment. 

In addition to obvious similarities between our 
legislation and other already existing programs, the 
Kentucky assessment has features which make it 
unique. Of particular note (and something you must 
certainly be ware of) is tieing the state-wide assess- 
ment to school accountabUity. That feature of the 
law makes this assessment different from other 
existing assessment programs. It also gives the 
Commonwealth an opportunity to lead other states 
as they introduce performance assessment 
programs. 

Issues of performance assessment in the Common- 
wealth are complex and exciting. Because the RFP 
has not yet been written and, obviously, a contractor 
has not been chosen, the precise form of the assess- 
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ment is not known. The structure of the assessment 
program, the content to be assessed and the methods 
of assessment are open in one sense. In another 
sense, there are some obvious precedents in the 
work of our task forces and existing practices m 
other states and countries. 

So, although there is ambiguity surrounding Per- 
formance Assessment in the Commonwealth, there 
are things we can do to prepare for program when 
it is implemented. 

We hope this conference will help you with that 
extraordinarily important work. Our morning ses- 
sions are designed to provide information about how 
performance assessments have been carried out in 
other arenas for various content areas and levels of 
schooling. We hope they will help you build a 
larger and more fully formed apperceptive mass 
about performance assessment. That apperceptive 
mass, a repetoire of information and insight will, 
we assume, help you respond to the assessment 
when it is precisely defined and implemented. The 
more we know; the better we will do. 

I said earlier that the committee has had an oppor- 
txmity to converse with those who have framed the 
law and those who at the state level are working on 
matters that are related to the assessment. It has not 
conversed with those in the schools who will imple- 
ment this program. Your second session of the day 
focuses on your insights and views about barriers to 
implementation. Those insights and views, along 
with other parts of today's program, will be simi- 
marized in a volume of proceedings. I will take that 
volume, containing those insights and views, to the 
committee. This your second task will, we hope, 
make the assessment committee's task easier. The 
committee welcomes your thoughts. 
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Performance Assessment: A National 
Perspective - Cautions and Concerns 



Gary W. Phillips 



Thank you very much fcr inviting me to speak 
at this exciting conference. There is much 
happening in educational reform all around 
the country and Kentucky is certainly the eye of the 
storm. The whole country is watching to see how 
your reforms turn out. They will certainly serve as 
a model for others to follow. 

I would like to talk about two of my favorite topics. 
The first topic will be NAEP and how your activities 
might piggy- back on what we are doing at the 
national level. The second topic will be what I think 
are some lessons I have learned in my work on 
performance assessments at the National level that 
you should keep in mind as you develop your 
performance based testing program. I want to warn 
you that Fm one of those evil psychometricians that 
your testing program will eventually have to deal 
with. 

First, let*s talk about NAEP. As you may know 
the NAEP has been around for over 20 years 
monitoring the educational achievement of our 
nation *s school children. We recently published a 
20 year summary of findings and the picture looks 
pretty bleak. The overall fmding is that in the past 
20 years there has been little or no improvement in 
any subject area, or at any grade. This is in spite 
of the large number of educational reforms and large 
increases in per pupil expenditures. 

In 1990, NAEP conducted its first Trial State 
Assessment in 37 States, two territorities and 
Washington, DC. The results will be released on 
June 6, 1991 in a national press conference and 40 
simultaneous State press conferences. Kentucky 
was one of those participating States. The results 
will cover mathematics at the 8th grade level for 
public schools only. In 1992 the Trial Sute Assess- 
ment will be extended to also include 4th grade 
reading and mathematics. Kentucky has also agreed 



to participate in 1992. Our National Assessment 
Governing Board is also setting educational stand- 
ards for the first time. When they are finished we 
will be able to provide State-by-State comparisons 
on how many students are achieving at a basic, 
proficient and advanced level. This will be the first 
time NAEP has gone beyond measuring what stu- 
dents know and can do to how many children have 
learned what they should know and be able to do. 

The Kentucky Education Reform Act requires you 
to pay attention to what is happening in NAEP. The 
Kentucky legislation requires the State to create an 
interim NAEP-Uke testing program beginning in the 
1991-1992 school year that is representative of each 
school. Let me discuss several ways this might be 
done. At the present time NAEP cannot be ad- 
ministered in such a way that student, school or 
district results are permissible. This is due to the 
prohibition in Federal legislation. First let*s talk 
about "What does NAEP like mean?" Fm not sure 
what you mean by the term but here is what I mean 
by the term. 1)NAEP objectives and item specifica- 
tions are used in developing your test. 2)Matrix- 
sampling is used such that each student in the class 
gets a portion of the test but no student takes all of 
the test. 3)Demographic and Contextual information 
is collected from students, teachers and principles 
to help explain student academic performance. 

4) The results are 'expressed on the NAEP scales. 

5) The test yields reliable information for schools 
and districts but does not provide reliable informa- 
tion for individual students. 

In spite of the legislation there are at least two ways 
that representative school type data can be obtained 
and still get national NAEP comparisons. The first 
would be to develop NAEP- like tests based on a 
combination of released NAEP items supple- 
mented by writing new NAEP-like items developed 
accordmg to the publicly available NAEP objectives 
booklet. As an example 3/7 of the 1990 Trial Sute 
Assessment items have already been released to the 
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public, and would be available for such a strategy. 
The link to the NAEP national scale would be 
possible through equating studies. The equating 
studies would be conducted (at the State's expense) 
in conjxmction with the NAEP assessment The next 
opportunity for this would be in the winter of 1992. 
A second approach would be to develop your own 
State testing program but have a portion of it repU- 
cate the NAEP. Again through equating studies at 
least that portion of your State testing program could 
be equated to NAEP. This would yield NAEP 
equivalent scores for a portion of your assessment 
while preserving the individual identity of your 
overall testing effort. 

Several commercial testing companies are looking 
into the profitability of both approaches and the 
University of Tennessee is seeking initial funding 
from several foundations to develop such proce- 
dures. These are at least two approaches to fulfilling 
your legislative mandate. There may be other crea- 
tive approaches. One thing to keep your eye on is 
the NAEP re-authorization that will likely occur 
next year. One of the issues Congress will deal with 
is that several State:? have passed legislation requir- 
ing their State testing to use NAEP, or NAEP-like 
tests. 

Let me now turn to the topic of authentic assess- 
ment. 

Let me begin by again confessing that I am a 
psychometrician. I also think Tm a good 
psychometrician. Psychometricians these days are 
like proctologists. No matter how good you are, 
you still get no respect. 

While standing in the hallway the other day I heard 
two experts on authentic assessment tell a joke. The 
joke was "what do you say about a psychometrician 
who is up to his neck in cement?" "More cement!" 

Since the beginning of the performance assessment 
movement, several years ago, psychometricians 
have gone into the closet. They have been branded 
as establishment conservatives who are more con- 
cerned with how the numbers come out than how 
the students come out. All around the country and 
even the world they are treated as the educational 
equivalent of lawyers. First kill die lawyers then 
we will have a better society. 

Let me begin my remarks by letting you know 
something about my experience with performance 
assessments. My more recent experience has been 
with the National Assessment of Educational 
Progress (NAEP). 



For 20 years NAEP has assessed writing using 
direct writing methods. We were certainly one of 
the first testing programs to assess writing in an 
authentic way. Our curren^'ifriting test requires 
students to write for 25 to 50 minutes and we score 
the test using analytic, primary trait and hohstic 
scoring methods. 

Our Adult Literacy Assessment in 1985-86 (to be 
repeated again in 1991) was also an entirely authen- 
tic test. There were no multiple-choice items. The 
test involved one-on-one interviews with the use of 
actual checkbooks, newspapers, bus schedules, 
maps, etc-. Our current Math assessment on which 
40 States and territorities are being compared invol- 
ves the use of rulers, protractors, and 30% of the 
items require scientific calculators. Thirty percent 
of the i^ems are open-ended and the entire assess- 
ment incorporated the NCTM views on how math 
should be assessed. 

Our upcoming reading assessment involves read- 
ing passages taken unedited from the hterature and 
40% of the students time is spent writing in response 
to open-ended questions. 

In past assessments NAEP used hands on tasks in 
science, assessed children's drawing skills, assessed 
students* ability to sing and play musical instru- 
ments, assessed dynamics of group decision-making 
and assessed students' ability to write computer 
programs. 

I mention these facts about NAEP because I hope 
to convince you that Pm a believer in the benefits 
and promises of performance assessments. Perfor- 
mance assessment strategies mav turn out to be the 
greatest advance in testing in recent decades. 
Several Sf ;es have taken the plunge (California, 
ConBv *cut, Vermont, Maryland and Kentucky) 
and repi — the leaders in the field. Performance 
assessments hold the greatest promise of assessing 
effectiveness of instruction, and keeping testing and 
curriculum aligned. However, I do think the 
benefits of performance assessments are more likely 
to be realized in classroom, district and possibly 
State level testing programs. As the scale of the 
assessment gets larger (national and international) it 
becomes more difficult (but not impossible) to real- 
ize these benefits. Although performance assess- 
ments clearly represent a major advance in the 
testing business, there are many exaggerated claims 
about their benefits, and many misconceptions that 
someone needs to talk about. When asked to speak 
at this conference I decided the most useftil service 
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I could provide would be to recommend some 
cautions based on our experience at the national 
level which will require your attention as you 
proceed with implementing a statewide performance 
assessment. I hope this will help to round out the 
cdvice you have been getting on the subject 

The first caution is that advocates of performance 
assessments argue that multiple«choice items cannot 
measure problem solving, reasoning and higher 
thinking skiUs. They cite evidence from NAEP and 
various international assessments that demonstrate 
that students are not learning these higher level 
skills. They argue that the reason students aren*t 
learning these skills is because we aren*t testing for 
them. But wait a minute folks. All of the evidence 
that students lack these skills is based on assessments 
that use primarily multiple-choice items. NAEP 
used multiple- choice items to show that in the past 
20 years we have improved in basic skills and 
dropped in higher order thinking. International 
studies that have come to the same conclusion use 
almost exclusively multiple-choice items. It is true 
that performance assessments possibly provide bet- 
ter measures of higher-order thinking skills, but 
don't forget that we also want tc Jcnow how many 
students only have basic skiUs. The jury is still out 
on how good performance assessments are at 
measuring basic skills, 

A second caution is that performance assessments 
are said to somehow lead to more eqmtable testing 
because minority students have a better chance to 
show what they have learned in a real world context 
which respects divergent ways of thinking, and 
creative solutions. Proponents of performance as- 
sessment envision a time when the skills of 
minorities will finally be tapped and the 
majority/minority gap us. educational performance 
will finally be closed. This may be true, I hope it 
is, but what does the evidence say? Based on 
preliminary data we have noticed the opposite in the 
National Assessment, The gap between White and 
Black students, and White and Hispanic students 
gets larger as the item format gets more authentic. 
In fact in some cases high proportions of the 
minority students get zero on ;ill of the more difficult 
authentic assessment items, 

A third caution is that many people argue that 
perforcjance assessments allow you to measure 
richer, fuller, more rewarding content. It is not 
always clear what this means but I think the argu- 
ment confuses two desirable but competing content 



considerations - depth versus breadth. Performance 
assessment allows you to measure a few things in 
great depth but more traditional methods will pro- 
vide a greater scope of coverage. Tradi^ 3nal 
methods of criterion- referenced testing achieve the 
greater breadth of coverage by using domain sam- 
pling methodology. In this method, the test is only 
a random sample of items from a larger domain 
(such as Geometry). By randomly sampling items, 
the test scores can be generalized to represent con- 
tent not included on the test. In performance assess- 
ments the concept of domain sampling is often 
abandoned. Here the focus is on the few good items 
that capture the big ideas, the thought processes , the 
actual skills you want the students to learn. In 
performance assessments, inferences beyond the 
items are tenuous. If the test includes the entire 
universe of things you want the student to know then 
you are in luck. However, in most cases it is not 
practical to test students on everything you want 
them to know. This is why domain sampling theory 
was invented in the first place. 

The third and I think the most important caution 
about performance assessments is that they are 
claimed to be more vaUd then multiple-choice tests. 
Supporters of this view cite the fact that authentic 
assessments sample the actual behaviors and skills 
we want students to learn. This is given as prima 
facia evidence of vaUdity, But this is what we 
psychometricians caU face vaUdity, or more skepti- 
cally, faith validity. You simply cannot tell how 
valid a test is by looking at it, or liking it. Nor can 
you make the test more valid by rhetorical exercises 
of creating new forms of vaUdity (such as ecological 
validity and boy-this-is-a-great-test vaUdity), 

The concept of validity is the most important 
concept in the testing business. People who claim 
they are testing professionals must understand this 
concept and must subscribe to the professional 
standards that define the concept and prescribe how 
it should be measured. Because the concept is so 
important I would like to spend much of my time 
talking about the concept and, if nothing else, I hope 
that I leave you with just one thing — a better 
understanding and appreciation of the importance of 
validity and an appreciation of how performance 
assessments must take the concept more seriously. 

Validity refers tc the extent to which inferences 
from a test are appropriate. It is not the test that is 
valid, it is the inference. The joint technical Stand- 
ards for Educational and Psychological Testing (by 
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AERA, APA, and NCME) define validity as the 
"appropriateness, meaningfulness, and usefulness 
of the specific inferences made from the scores. 
Test validation is the process of accumulating 
evidence to support such inferences ... validity is a 
unitary concept (and) ... always refers to the degree 
to which that evidence supports the inferences that 
we made from the scores. The inferences regarding 
specific uses of a test are validated, not the test 
itself^ (P'9). So in what sense can performance 
assessments be more valid than multiple-choice 
tests? They cannot be more valid in the sense of 
greater face validity because face validity is not a 
form of validity. Face validity is a ccxicept outside 
the joint technical standards and refers to the politi- 
cal acceptance or the appeal of the test based on its 
superficial appearance. It is abundantly clear that 
performauce assessments have greater face vaUdity 
then multiple- choice tests. However, that doesn't 
mean they are more valid in the sense of providing 
better inferences. 

One way in which performance assessments might 
be argued to be more valid is that they are more 
representative of the defined universe or domain of 
content we wish to test. That is they have greater 
content validity because they better represent the 
actual skills and behaviors we want to learn. They 
are samples of the most important parts of the 
curriculum on which there is expert consensus, and 
what we want teachers to emphasize. 

Although performance assessments hold the 
promise of being more content valid, the evidence 
for such validity is often asserted but seldom 
demonstrated. To be content valid means that there 
is a universe of content to which we wish to make 
inferences from the student's test scores. For ex- 
ample, we may wish to make an inference about a 
student's narrative writing skills from his score on 
a writing prompt, or we may '^ish to make an 
inference about a student's understanding of the 
scientific method from a science projf^ct. But how 
can these inferences be valid if the writing skills of 
hundreds of thousands of students are based on only 
one or two writing prompts (a one or two item test 
- the norm for virtually all performance testing 
programs) and how can you compare hundreds of 
thousands students on how much they understand 
the scientific method if each is doing a different 
science project? The issue is not that hundreds of 
thousands of science projects can be evaluated by 

trained evaluators with rigid rating criteria. The 



issue is whether the inferences are equally valid for 
each of the hundreds of thousands of students. If 
they are not, then the testing program is imfair to 
students for whom it is less valid, especially if high 
stakes decisions, such as promotion or graduation, 
are a consequence of the scores. 

The problem of equal validity is exacerbated when 
alternate forms of the testing instrument must be 
developed in subsequent years. If a reading assess- 
ment uses a well known short story such as "A 
Tell-Taie Heart" by Edgar Allan Poe, then teachers 
will start teaching the short story* Since we want 
to teach lots of short stories (not just the one on the 
test) we must use a different short story on the test 
the following year. But it is hard to find another 
short story which has all the properties we liked 
which led us to select "A Tell-Tale Heart" in the 
first place. And what about the next year, and the 
next. If we choose short stories that are different 
form "A Tell-Tale Heart" it is unfair to measure 
subsequent students by a different yardstick. This 
is the psychometric problem of equating in authentic 
assessments. Anyone can conduct a good assessment 
once, the real trick is to do it twice. 

In the final analysis the greatest obstacle to the 
validity of authentic testing is unreliability. Un- 
reliability ;s the Achilles' heei of performance as- 
sessments. Eventually the problem must be dealt 
with in a responsible way. We are wrestling with 
it at the national and international level, and Ken- 
tucky will have to deal with it at the State level. Let 
me take a Hiinute to explain what I mean. There is 
a mathematical relationship between the reliability 
and the validity of a test. It is impossible for the 
validity to exceed the square root of the reliability. 
This means that an unreliable test cannot possibly 
be valid. There are two reasons why performance 
assessments are generally less reliable than multi- 
ple-choice type tests. 

The first is that authentic tests typically employ 
fewer items. Because of the cost of scoring and the 
time burden on students, most authentic assessments 
employ so few items that their reliability is suspect. 
Some State assessments of writing use only one 
item. Under these circumstances the reliability of 
the test cannot even be estimated (this requires at 
least two items). 

The second major reason why performance assess- 
ments tend to be less reUable is because they cannot 
be administered under standard conditions and they 
cannot be objectively scored by a computer. The 
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administratioD and scoring of the assessment permits 
many sources of xmcontroUed variation. The test is 
administered under more realistic real-life, uncon- 
trived conditions. The students' responses are 
scored by raters who vary in their grading leniency. 
To better understand this it is helpful to imagine an 
analogy where performance assessments are used 
routinely. We often hear that performance assess- 
ments are similar to what a football coach does to 
evaluate a player. But in a large scale assessment 
the analogy is more complicated. Imagine thousands 
of football coaches evaluating hundreds of 
thousands of players, using thousands of different 
plays which are chosen by the players, not the coach. 
This is identical to a portfolio assessment or an 
assessment of science projects. With so many un- 
controlled variables it becomes very difficult to 
make objective (i.e., equally reliable and valid) 
decisions about the students* performance. 

I would like to end my talk with some free advice. 
My advice is basically that Kentucky needs to spend 
a lot of time understanding the practical and techni- 
cal measurement problems associated with starting 
a new, innovative testing program. The program 
will affect the lives of countless thousands of stu- 
dents, parents and teachers. Furthermore, you are 
doing it in a fish bowl. The whole country is 
watching to see if you can pull it off. You've made 
the decision thai you are go^-^p to the moon. Now 
everyone is watching to see if you get there. I have 
five specific recommendations. 

Recommendations 

• Think through measurement issues. Ken- 
tucky has spent a lot of time and resour- 
ces focusing on curriculum and instruc- 
tion issues. Spend some time on meas- 
urement issues. After all testing is a 
measurement activity. A large scale test- 
ing program must address hundreds of 
technical and theoretical measurement is- 
sues that go beyond the curriculum. 
You must build a testing program that 
responsibly addresses these problems. 
Some of the technical issues include test 
bias, scaling, cost, standardized ad- 
ministration, reliability, validity, equat- 
ing, and scoring. The theoretical issues 
center around the purpose of testing. 
What are you trying to do? Are you 



trying to monitor performance, trying to 
improve performance, hold teachers and 
their schools and districts accountable, 
make decisions about resource alloca- 
tions, evaluate programs, promote or 
graduate students, give diagnostic feed- 
back to teachers, evaluate instruction, 
evaluate curriculum? One test cannot do 
all of these things. Decide up front 
what you are trying to do. 

• Read carefully and conform to all of the 
joint technical standards written by 
APAj /r.i^ and NCME. These repre- 
sent the best thinking by the best people 
in the field about your professional test- 
ing responsibilities. Your testing pro- 
gram must meet and hopefully exceed 
the standards of the profession. When 
your actions start to actually impact 
teachers and students you will need the 
professional standards to corroborate 
your procedures. If there are legal chal- 
lenges you will have support for your ac- 
tions. If you do not follow the joint tech- 
nical standards then I suggest you keep 
your resume current. 

• Now my most heretical reconmienda- 
tion. Don't completely abandon the mul- 
tiple-choice format. Try out the new 
and more innovative approaches to as- 
sessment, but build in an honest evalua- 
tion system that will help you make ap- 
praisal of your success, and give your- 
self time to make mid-course adjust- 
ments. Take a risk, but remember the 
fundamental economic principal in any 
investment - the greater the risk the 
greater the need for diversification. Try 
out performance assessments. But don't 
forget the lowly multiple-choice item. It 
has been around for over a hundred 
years performing the thankless task of 
providing reliable and valid data in a 
cost effective maimer. I suggest you in- 
clude it in your assessment portfolio. 

• Get the very best people to work on the 
project. Not only in its initial stages, 
but even more importantly, for the long 
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haul. Long after the visionaries have 
gone on to other callings, long after the 
consultants and contractors have set the 
system in place, there will be a gro\xp of 
nameless, faceless, but loyal civil ser- 
vants who like the multiple- choice item, 
will be called upon to make the system 
work. 

• Finally, don*t be deterred by people like 
me who say it cannot be done. It can be 
done. But be prepared for a difficult 
task. The last time we oaid we were 
going to the moon we made it. Even 
though it was a complicated technologi- 
cal task the basic science was in place to 
get us there. What was needed was the 
will and resources. We are in an identi- 
cal situation with performance assess- 
ment. The will is there, the science is 
there. What we need now are the effort 
and the resources. What we have 
learned at the national level is that per- 
formance assessments involve a lot of 
people, take a long time and cost a lot of 
money. If Kentucky is willing to pro- 
vide the effort and resources then I wish 
you well on your way to the moon. 
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We began the session by discussing the 
following definitions of performance as- 
sessment and the criteria for high quality 
performance tasks currently being used by the 
Council on School Performance Standards to guide 
the task forces in the development of performance 
tasks for Kentucky schools: 

Performance-Based Assessment is defmed as 
giving a group of students, or an individual stuaem, 
a task that may take from half an hour to several 
days to complete or solve. The object of the assess- 
ment is to look at how students are working as well 
as at the completed tasks or products. An observer 
or interviewer may stay with the group or make 
periodic visits. Activities may be videotaped, tape 
recorded, or recorded in writing by an adult or 
students. The task might be from any content area 
and might involve any subject (Lee and Lee, 1990). 

Roger Pankratz (1990), Executive Director of the 
council of School Performance Standards, states that 
"performance assessment measures a student's 
ability to perform a task like writing a business 
letter, balancing a checkbook, performing a science 
experiment, using a word processor to edit a 
manuscript or repairing an appliance. Performance 
assessment usually involves measuring a student's 
ability to use knowledge and skills to produce an 
outcome or product . . . [and thus] looks beyond the 
knowledge and skills we usually measure with paper 
and pencil tests to what students can do wi.h the 
knowledge and skills they have acquired." 

Drawing upon the work of Grant Wiggins, 
Pankratz has suggested that the following criteria be 
used to evaluate the performance tasks currently 
being developed in Kentucky: 



• Essential vs. Tangential - The task fits 
into the core of the curriculum and repre- 
sents the "big ideas" of the discipline. 

• Authentic vs. Contrived - The task as- 
sesses skills, knowledge, and processes 
that are most appropriate to, valued in, 
and commonly used in the discipline. 
The task is worth learning and teacning 
and simulates the work of adults in a 
given field of study. 

• Rich vs. Superficial - The task leads to 
other problems and raises other ques- 
tions; it is "pregnant with possibilities." 

• Engaging vs. Uninteresting - The task 
is thought-provoking, chaUenging and 
fosters persistence. 

• Active vs. Passive - The task en- 
courages student activity and interaction 
with other students and involves the stu- 
dent in constructing meaning and deepen- 
ing understanding. 

• Feasible vs. Infeasible - The task can 
be performed with available resources 
and completed within reasonable time 
constraints. 

Assessment tasks which exemplify these criteria 
will drive instruction in positive ways. They will 
encourage teachers t focus on integrated applica- 
tion of important learnings rather than on memoriza- 
tion of isolated skills. They will encourage students 
to think through the processes they are using and to 
explain why and how they achieved results. 

Fortunately, our best teaching practices are al- 
ready preparing students for this type of assessment. 
Examples of these practices include integrated, 
holistic reading and language arts instruction; writ- 
ing across the curriculum; hands-on, manipulative 
experiences in mathematics and science; and 
cooperative learning. 

We discussed Goals 3 and 4 of the Six Learning 
Goals (Council on School Performance Standards, 
1990). Goal 3 stresses Self-Sufficiency and Goal 4 
stresses Group Membership Skills. Most par- 
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dcipants agreed that the best way to assess these two 
goals is within the context of the tasks designed to 
assess Goal 1 Basic Skills and Goal 2 Core Con- 
cepts. Many of the desired outcomes related to 
self-sufficiency and group membership (e.g., learns 
on his/her own and recognizes the viewpoints of 
others) could be observed while students were per- 
forming either groiq> or individual tasks in the basic 
content areas. However, some members of the 
group expressed concerns about whether group 
dynamics could negatively affect the evaluation of 
individual performance, and might interfere with the 
evaluation of the student's understanding in a given 
content area. 

Teachers will need to learn how to facilitate 
cooperative learning groups if students are to be 
prepared for working effectively in group situations. 
This was viewed as another example of one of the 
ways in which the new performance assessment will 
drive instruction in positive ways and provide affir- 
mation for the teachers who are akeady engaging in 
Innovative teaching techniques. 

All participants agreed that the criteria for a good 
performance should be agreed upon in advance and 
shared with the students ahead of time. It is impor- 
tant that both teachers and students be made aware 
of what constitutes an expert performance on a given 
task so that the teachers will be able to prepare 
students for the task and students will be able to 
engage in self-evaluation. 

Participants expressed strong concerns about the 
validity and reliability of performance assessment. 
Although the individual tasks have high face 
validity, time constraints will make it impossible to 
sample broadly from the total discipline because 
each task will take a long time to administer. For 
example, if students are tested only on one or two 
tasks related to history, will the results of that 
assessment provide an accurate reflection of the 
students' historical understanding? If there are 
strong within-student differences among tasks in a 
content area, the results may present a misleading 
picture. Will the trained assessors be able to make 
reliable judgments after observing students for only 
a short time in a limited number of situations? 

While we agreed that authenticity certainly con- 
tributes to the fiace validity of a task, there is a danger 
that tasks that are truly authentic in one geographic 
area of the state might not be authentic in another, 
"^or example, a math question regarding toll booths 
might not provide a familiar context for students in 



Eastern Kentucky who have not traveled in the 
western par*- of the state. 

Furthermore, there were questions regarding the 
matrix sampling procedures to be used in selecting 
the students from a given school who will be asked 
to participate in a particular performance assessment 
task. Will enough students be assessed to give a true 
picture of the performance of the total school in a 
given discipline? 

Several questions were raised about the Interim 
Test to be administered in 1991. If the Interim Test 
is to resemble the NAEP (National Assessment of 
Educational Progress) tests rather than the Kentucky 
Performance Assessment Tasks, how will the state 
be able to set the threshold level on the Interim Test 
and then judge school performance by a retest on 
the Kentucky Assessment? Will some schools pur- 
posely score low on the Interim Test so that they can 
show improvement on the next testing? As soon as 
the state has made decisions regarding the Interim 
Tests, educators need to be informed immediately 
to enable them to prepare for the assessments. 
Presently, anxiety levels are high. 

Overall, participants felt that the positive effects 
of performance assessment will likely outweigh the 
negative in that it will drive instruction in positive 
ways and keep the focus on learning processes and 
authentic products rather than on isolated skills and 
will affirm the efforts of our most innovative 
teachers who are abready stressing cooperative 
learning and developmentally appropriate cur- 
riculum. However, participants expressed strong 
needs for more information about the nature of the 
performance assessment to be used in Kentucky and 
for staff development to help teachers learn how to 
prepare their students for this new type of assess- 
ment. 
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Afternoon Session 

A. Do the activities defined, illustrated and dis- 
cussed in the morning session meet the requirements 
of the law? If not, what must be done to create a 
fit? 

Due to the fact that the afternoon participants had 
attended different session during the morning, there 
was little common ground in this session for answer- 
ing this item. Nevertheless, the discussion revealed 
that neither participants nor leaders could list with 
certainty the requirements of the law in the area of 
performance assessment. 

Lack of clarity and specification can be positive as 
well as negative. It is often part of any new en- 
deavor and can be an opportunity for individualiza- 
tion and creativity. However, our discussion 
indicated that in this instance the lack of clarity also 
involves a lack of direction and basic knowledge 
regarding performance assessment. 

As difficult as it may be, those individuals dire^n- 
ing performance assessment in Kentucky need to 
concretize guidelines regarding what performance 
assessment means La the Commonwealth. This 
statement is made with full recognition that a final 
determination may only be possible with the evolu- 
tion of the product. Nevertheless, as soon as pos- 
sible, everyone involved with public education in 
Kentucky needs to be provided with basic 
knowledge, guidelines, definitions and samples 
regarding tho range of what performance assessment 
is and what it may be like in the Commonwealth. 

B. What are the structural barriers, if any, to using 
these activities in the schools? 

Session participants listed the following potential 
barriers: 

• Use of textbooks which do not facilitate 
the preparation of students for perfor- 
mance assessment. 

• Inadequate preparation of older students 
to integrate material from different sub- 
ject areas. 

• Short length of time for implementation 
of the new approach to assessment. 

• Task content 

> Who will select tasks? Will there be 
input from teachers? 

> Will there be an appeal process for 
districts/schools who disagree with 
the content, scoring, criteria, etc. of 
a task? 



• Sampling 

> How will the matrix be developed 
and implemented? 

> What steps will be taken to insure 
that samples are not biased? 

Participant discussion of these barriers revealed 
the high level of anxiety regarding performance 
assessment felt by both teachers and administrators. 
Participant statements indicated that much of this 
anxiety was due to a lack of knowledge regarding 
ways to begin to prepare for the new assessment. 
Thus, the importance of training, a topic which is 
discussed more thoroughly below, was clealry indi- 
cated by this group's discussion of barriers. Indeed, 
this group's discussion suggested that a lack of 
adequate training would be a major barrier to the 
implementation of performance assessment. 

C. What are the Costs of Implementing these new 
Activities? 

Our group did not respond to this question in terms 
of dollars and cents. However, they did indicate 
that costs will occur in several areas including test 
development, test administration, school operations 
and school functioning (hidden). 

In regard to the tests themselves, the development, 
pretesting and revision of the assessment tasks will 
require extensive preparation and work. To carry 
out the state wide administration of the performance 
assessments, a large number of full-time personnel 
must be hired. These task administrators should be 
professionals hired at professional salaries. The 
administration of the tasks will also require a large 
travel and materials budget to enable the task ad- 
ministrators to do their job. 

School related costs will include a budget line to 
support training (see below), and one for the pur- 
chase of materials, texts and books which will enable 
teachers to be effective as they teach for a new type 
of assessment. Hidden costs for schools may in- 
clude the loss of some good teachers who are near 
retirement. Schools may also incur stress related 
costs such as an increase in sick days, low energy 
levels among faculty and administrators and an 
increase in the number of staff disputes. 

D. Are There Areas in which Training Will Be 
Required? 

The need for training was evidenced in almost 
every area discussed during the afternoon session. 
We cannot begin to implement the requirements of 
the law until we understand how performance as- 
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sessment will be defined in the Commonwealth. 
Both teachers and administrators must become 
familiar with the nature of performance assessment 
in general and performance assessment in Kentucky. 
(Training in the former can be done right now, 
training in the latter should commence as soon as 
possible.) 

Teachers also need training which will help them 
understand what they can do in the classroom to 
prepare their students for performance assessment. 
Part of this training can focus on simply reminding 
teachers of techniques with which they are already 
familiar. Some of the suggestions which were made 
during our afternoon session were a) think of ac- 
tivities which motivate students and help them to 
learn, b) to talk to teachers who have hands-on 
activities in their classrooms, c) adapt performance 
based approaches to education, such as Montessori, 

d) use the principles of teaching for transfer, and, 

e) mentor and learn from new teaches with creative 
ideas. 

A third group which will need training regarding 
performance based assessment is parents. If they 
are to support their children and the schools, they 
need to be aware of the manner in which their 
children will be evaluated and the skills which their 
children need to develop. This training should be 
introduced during the year preceding the first per- 
formance assessments. 

It is important that school councils /administrators 
recognize and support the need for training. This 
will require the allocation, for both faculty and staff, 
of funds and time for inservice sessions and atten- 
dance at meetings and workshops. Each dis- 
trict/school must also determine the best way to 
educate parents, and insure thai as many parents as 
possible are informed regarding performance as- 
sessment. Districts/schools should also be aware 
that training in the areas of communication skills 
and the management of stress may be very helpftil 
during the implementation of KERA. 

Finally, it is extremely important that state and 
regional education related organizations include ses- 
sions regarding performance assessment during 
their annual meetings. These meetings are affor- 
dable and may provide one of the most cost effective 
ways to provide training. 

E. Session Simmiary 

Participants in the afternoon session on elementary 
education clearly indicated the need for clarity 
regarding performance assessment in Kentucky. 



Barriers to the implementation of performance as- 
sessment include time and means to prepare students 
for a new type of assessment, task content, ad- 
ministration sampling procedures and generalized 
anxiety. Two keys to minimizing the barriers and 
successfully implementing perfonnance assessment 
are training for faculty, administrators and parents, 
and adequate funding for all aspects of the program. 
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The Kentucky Education Reform Act of 1990 set 
the following goals for the Commonwealth's public 
schools: 

1. Schools must expect a high level of achieve- 
ment of all students. 

2. Schools must develop their students' abihty to: 

• use basic communication and mathe- 
matics skills for situations they will en- 
counter throughout their hves; 

• apply principles from mathematics, 
sciences, arts, humanities, social studies 
and practical living studies to situations 
they will encounter throughout their 
lives; 

• become self-sufficient individuals; 
become responsible members of a fami- 
ly, work group or community; 

• think and solve problems in school situa- 
tions and in hfe; 

• connect and integrate experiences and 
new knowledge with what they have pre- 
viously learned, and build on past learn- 
ing experiences to acquire new informa- 
tion through various sources. 

3. Schools are to improve their students' rate of 
school attendance. 

4. Schools are to reduce their students' dropout 
and increase retention rates. 

5. Schools are to reduce physical and mental 
health barriers to learning. 

6. Schools are to be measured on the proportion 
of students who make a successful transition to 
work, post-secondary education and the military. 

(KERA, 1990). 



One aspect of KERA charges the State Board for 
Elementary and Secondary Education to develop a 
statewide assessment program, including perfor- 
mance-based student testiag, to measure the success 
of each school according to the above goals (Legis- 
lative Research Committee, 1990). Students, 
beginning in the 1993-94 school year, will be tested 
in reading, mathematics, writing, science, and so- 
cial studies in grades four, eight, and twelve. Local 
school boards wiU be required to pubhcize an annual 
report of performance of students. Thus, elemen- 
tary schools will be expected to do performance- 
based assessments of their students. 

What i<; performanc^e assessment? What exactly do 
we mean? 

Performance based assessment measures a 
student's ability to use knowledge and skills to 
perform a challenging activity or tai»lc. This imphes 
that the assessment looks beyond mere knowledge 
and skills to actual application. Performance assess- 
ment requires a set of criteria for evaluating the 
quality of a performance (Wiggins, 1989: Pankratz, 
1990). The attributes for the set of criteria include 
essential, authentic, rich, engaging, active, and 
feasible (Connecticut State Department of Educa- 
tion, 1990). Methods for the evaluations can in- 
clude performance tests, observations, exhibitions, 
interviews, and portfolios (Mitchell, 1989). 

By its name, assessing performance implies stu- 
dents performing, or "doing" something. What 
students "do" should be tasks that approach real-hfe 
situations. To assess these tasks seems to require 
some type of observations. But is it necessary to be 
observed? Are certain skills even observable? If 
the tasks are to be apj^ropriate, the tasks could thus 
require a lot of time to complete. Will not this time, 
therefore, present more problems? Who will do the 
observations? How wUl they observe? What will 
be the criteria? 

Along with observations, student portlolios also 
seem to be appropriate to assess performance. Here 
again, logistics will be a strong concern. How will 
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these portfolio records be compOed, produced, and 
stored? Also, what will be required of the content 
of the portfolios? 

Performance-based assessment lends to be viewed 
as either promoting convergent thinking or diver- 
gent thinking, depending on the point of view of the 
educator. If, in order to assess performance, scor- 
ing rubrics are stated, how will creativity and 
resourcefulness be included? Upon whose criteria 
will the scoring rubrics be constructed? 

What are the criteria for det ermining whether 
performance a5>sessments are go od or bad? When 
comparing performance aj;fie5;5;mpntg tn more tradt^ 
tional paper and pencil meajoires are there issues of 
validity or efficiency? 

For "authentic" performance-based assessment, 
logistics can be a major concern with issues of 
validity. Reliability will be difficult to obtain. For 
example, in cooperative group situations, how will 
standard conditions and standard criteria be pos- 
sible? Will the evaluators be able to distinguish 
knowledge, basic skills, and performance? As to 
efficiency, feasibility concerns will determine the 
type, the quaUty, and the quantity of the assessment 
in a school environment. Finally, what will be the 
turn-around time between assessment and the 
teacher obtaining the results? 

assessment, in what sense have they v/orked well? 

Some areas of the curriculum are already perfor- 
mance-based, e.g., physical education and vocation- 
al education. However, these activities do not go 
to the extent desired for authentic, engaging, and 
rich criteria. The approaches of literature-based 
language arts can lead to integrated thematic units 
in the cuiriculum. These current forms require a 
knowledgeable, skilled teacher, dedicated to the 
goals of the program. The teacher, therefore, must 
become a risk-taker, one willing to try new ap- 
proaches in instruction and assessment. 

Are there implications of performance assessment 
in terms of fairly assessing the different genders, 
racial and ethnic groups, and social classes? 

In Kentucky, the Appalachian population is a 
major concern. Will the "language" utilized in the 
assessment be appropriate for this "minority? There 
is the possibility that low assessments may represent 
the language imfamiliarity of the assessment, rather 
than low performance. Thio same concern can 
occur for the inner-city African American popula- 
tions of the state. Students* environmental and 



cultural background will indeed have an in^>act on 
their performance. The assessment will need to 
make sure this impact is not a negative one. 

Finally, there is the unresolved problem of the 
special needs populatiops. Mainstreaming students 
can fluctuate a classroom or a school's's assessment. 
Special logistics will need to be standardized for 
these students. 

Do the activities defined. illiLstrated , and discussed 
meet the requirements of the law? If not, w hat must 
be done to create a fit? 

For whole state feasibihty, matrix sampling utiliz- 
ing scoring rubrics will be required. School districts 
will need to derive their own assessments for non- 
fourth, eighth, and twelfth grades. Districts will 
thus have the responsibility, but not necessarily the 
rights of decision making, for performance-based 
assessment. Districts, in a sense, will need to "get 
in line" with the state assessment criteria and pro- 
cedures for their own assessments. An appeals 
process will need to be in place for districts and 
schools to use if the institutions disagree with the 
state results or criteria. 

What are the structural barriers, if any to using 
these activities in the schools? e.g.. Are schools 
prepared to implement these new kinds of testing 
procedures and can they do it routinely? 

After the assessment is changed, the next physical 
barrier will be the classroom textbook. Unfor- 
tunately, a majority of teachers tend to rely on their 
texts as a curriculum guides for instruction. The 
type of instruction envisioned by performance-based 
assessment differs from the skill-and-drill textbooks 
-currently available. The monetary, political in- 
fluence of the textbook industry may try to intervene 
in this area of school reform. 

Another major barrier will be the teachers. Cur- 
rently teachers are poorly informed of the exact 
details of the performance-based assessment they 
implement. Teachers, as well as administrators, 
will need to obtain "ownership" of the assessment 
if positive change is to occur. 

What are the costs of implementing these new 
activities? (^Materials as well as labor;^ 

After state assessment instruments are developed, 
the major expense will be staff development. 
Teachers and administrators will need to be in- 
formed of, then included in, the implementation 
stage of the assessment. Inservices will need to be 
developed and presented where educations can 
develop, test, and evaluate their own types of per- 
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fommce assessments so to be comfortable with this 
new area of education. 

The reform act provides monies for the state 
assessment, but who will provide the cost of 
developing and implementing district-level assess- 
ments for the other grade levels? If continuous 
assessment is to be obtained, will not a continuous, 
on-going cost occur? 

Are there areas in whic^b training will be required? 
What areas and what training^ 

Following staff development for the development 
and implementation stages of the assessment, even 
more inservice assistance will be needed in the area 
of instruction. The curriculum performance-based 
assessment reinforces is much different than the 
current scope and sequence. Many teachers will be 
asked to instruct in a manner different than their 
current practice of lecture and individualized prac- 
tice. Teachers cannot be expected to "reform" their 
practices without assistance and without support. 

The national professional organizations, e.g., Na- 
tional Council of Teachers ofMathemati.es, already 
have taken a lead in the development of curriculum, 
instruction, and assessment methodology that can 
aid the classroom teacher. The problem is the 
inservices dealing with the state reform act are using 
up all available professional days and monies. 
Teachers are having difficulty attending and par- 
ticipating in the national organizations because of 
the financial and time requirements of KERA. Cur- 
rently, the professional organizations are competing 
with KERA instead of complementing it. 

What are the relationships, if any, among perfor- 
mance assessment, school-based deci<;inn making 
and other aspects of TCFRA? 

The aspects of KERA are broad and wide-reach- 
ing. Education in Kentucky will need to change 
dramatically to conform to KERA*s requirements. 
By changing the entire system, instead of a 
piecemeal approach, educators are now aware, even 
excited, about the reforms. So the first two steps 
toward attitude change are occurring, namely, 
awareness and commitment. 

For substantial change to occur, educators and the 
public will need financial and logistical support. To 
this end, the various aspects of the reform act may 
impede each other. Just so ruach staff development 
and curriculum development can occur during a 
given time interval before all change is rejected. 
Careful watch must be kept to make certain all the 



reforms do not overload the education system nor 
the educators. 
Summary 

More than any aspect of the Kentucky Education 
Reform Act, performance-based assessment will 
change, hopefully reform, the curricula in the 
schools. To some teachers in the Commonwealth, 
the assessment will be a reward for the quality job 
they have been doing for a long time; to some, the 
assessment will be a punishment to force long 
overdue changes in school content and method; but 
to all educators, the assessment will direct the daily 
classroom instruction towards a more reaUstic, prac- 
tical, applicable, and yet rigorous level. To the 
public, assessment is a way to obtain accountability 
of the use of precious resources — our time, our 
money, our children,- our future. Yet to many 
educators, assessment is viewed as a possible 
punishment — it is feared. Teachers must be totally 
involved in the design, insullation, and evaluation 
of performance-based assessment for it to be suc- 
cessful. How the new assessment in constructed, 
implemented, and then utilized will determine if 
performance-based assessment will obtain its poten- 
tial as a beacon to guide education in Kentucky. 
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Approximately 28 energetic school personnel 
from around the state gathered to learn new 
information about the Kentucky Education 
Reform Act of 1990. They were asked, "What does 
performance assessment mean to you? Their 
responses were in the form of statements that could 
be made and questions that still must be answered. 
These responses may be grouped as follows. 

Broad definitions: 

What do you know and how do you show it? 

Demonstrating that a student can apply what has 
been learned. 

Interdisciplinary testh?g across subject areas. 

Correlation of goals and objectives and how well 
they are met. 

Characteristics of performance 
assessment: 

Performance assessment involves random sam- 
pling; practical applications; and multiple-choice 
questions. 

Writing samples may be included. 

Multiple instruments may be required. 

Performance assessment is integrated across all 
areas of the curriculum. 

It includes basic knowledge, process skills, and 
higher order thinking skills. 

Performance assessment involves conceptual 
teaching and a focus on individual styles and needs. 

Concerns about assessment: 

Are students and teachers ready to move to perfor- 
mance-based assessment? 

There may be more than one right answer to some 
test questions. 



Requires a change in evaluation procedures. 
Caution is needed. 

Special populations, such as the gifted, wiU need 
special consideration. 

There is a need to lay the groundwork for teachers. 
They may need training if they are unfamiliar with 
this method of assessment. 

Performance assessment involves the evaluation of 
the school rather than the individual. 

There are many questions now, but few answers. 

Are we going to assess abihty or what was taught? 

There is a problem in comparing schools. 

Is this just another reform wave? 

How will local assessment be affected? 

It seems like we are painting an old house with 
new paint. 

There is an appearance of two testing systems— one 
NAEP and one performance-based. Are these the 
same? 

Cost is a concern. 

How will these tests be used? 

How will feedback from the tests be used? 

It will require a change in teaching emphasis. 

We will need to help teachers with it. 

Additional work will be required beyond what we 
are already burdened with. 

There will be an effect on the dynamics of the 
classroom. 

There are questions about the threshold. Should 
we start low so we can be assured of progress? 

We will still need a measure of individual perfor- 
mance, especially for parents. For example, the 
CTBS may be used. 

The ability range of students is wide. 

There is a lot to do in a small amount of time. 

Grading turnaround will be too long. For ex- 
ample, NAEP tests require a year to be OTaded. 

The afternoon session included 16 diverse school 
personnel also in search of answers about student 
performance assessment under KERA. They were 
first asked for a working example of performance 



16 



Middle School 



assessment. Among the ideas shared were the 
following. 

A scenario involving shoe sales entitled "Who wins 
the shoe war? " which could incorporate both math- 
ematical and communication skills. 

The debate over the right to bear arms addressing 
both the Constitution and communication. 

A job application with connections to life, prob- 
ability games, and self concept 

The use of the class as a student council for group 
dynamics. 

Recycling as an application project. 

Family economic awareness involving a budget 
and the cost of raising a family. 

The group was next asked, "What are the criteria 
for good performance assessment? Their respon- 
ses are listed as follows. 

It should: 

be interdisciplinary. 

be a synthesis of the students* learning. 

include conceptualization and apphcation. 

be reliable. 

be clear. 

be continuous. 

be appropriate. 

be facilitative. 

be fun. 

include cooperative learning, 
be creative. 

The third question posed the group was "What are 
examples of current practices in this area which 
have worked well?" Their responses included the 
following: 

The Writing Prqject 

Portfolios including writing samples and hohstic 
scoring which can be passed on to other teachers in 
subsequent grades. 

Manipulative math 

Hands-on science 

Science-math integrated projects 

The fourth question was "What new information 
is needed relative to student performance assess- 
ment?" Their answers included: 

People can get excited about student performance 
assessment! 



We're all in it together and can learn together what 
shape it will take. 

There is an opportunity for ownership from 
teachers. 

There will be a need for teacher inservice ac- 
tivities. 

How will it change the schools and students? 

Are we a sounding board? 

We are asking questions but not receiving answers. 

There will bft a lot of power in the hands of 
textbook publishers. 

What is the relationship of performance assess- 
ment to other areas? 

There are many different ideas about performance 
assessment. 

There is a need for what the questions are going 
to look like, for example, in math. 

There are a number of test issues, for example, 
problems with validity and reliability. 

There is a need for input from practitioners. 

Higher education is enthusiastic about helping 
direct assessment. 

There is a need to continue doing what we are 
already doing right. 

We need to involve more teachers. 

Finally, the group was asked to identify barriers 
and facilitators related to student performance 
assessment. Among the barriers listed were: 

Mind set of seasoned teachers 

Validity of performance based assessment 

Who are the evaluators? 

Who*s in control? 

Time to do the assessment 

Time, people, and money 

Fear, accountability 

Geographic bias 

Setting parameters so can develop die ciuriculum 
Funding 

Leveling off point 

Interdisciplinary curriculum planning, staff 
development, and instructional strategies were 
viewed as facilitators. 
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Reading, Writing and Language Arts 



Peter Winograd 

Jim Craig 

What is performance assessment in read- 
ing and writing? What are the criteria 
for deciding whether they are good or 
bad? Are there workable examples of performance 
assessmentix in the language arts? How can we 
implement performance assessments in the language 
arts in Kentucky's schools? What kinds of training 
will be required? These and other questions 
provided the impetus for spirited discussion and 
debates among the teachers, administrators, and 
university professors who attended our sessions. 

We came to some conclusions during our discus- 
sions, but we raised more questions than we 
answered. Perhaps the most important lesson 
learned was that many of our concerns were shared 
by others. We will present the results of our 
deliberations in a series of interrelated points. 

1. Performance assessment in reading, writing, 
and the other language arts must focus on important 
outcomes rather than on fragmented or isolated 
skills. Some of the important outcomes identified by 
our discussion groups included the ability to com- 
prehend what has been read; the ability to write 
clearly for a variety of authentic purposes; a love of 
reading; the ability to use reading and writing to 
grow in affective, personal ways; and the abUity to 
apply reading, writing, and the language arts effec- 
tively across a wide variety of subject areas. 

2. A great deal of our discussion focused on what 
it means to comprehend. We believe students should 
be able to read authentic texts and, depending on the 
type of text and the purpose for reading, display a 
global understanding, a more detailed under- 
standing, a personal response, or a critical evalua- 
tion. 



3. Participants in our sessions expressed their 
hopes that writing would become more common in 
many subject areas and not just in English. For 
example, writing in response to what is read in social 
studies or science is as valid as performance assess- 
ment task as is writing in response to what is read 
in literature. This statement, along with points 1 and 
2, reflects the importance of ensuring that the 
language arts are integrated with other subject areas. 

4. Performance assessment in reading, writing, 
and the lang\iage arts should include authentic texts 
and entail a variety of response fonnats for students. 
Students can display their understanding in a number 
of ways including writing in response to reading; 
discussing what they have read; working in groups 
to extend and communicate their understanding of 
what they have read; following or composing direc- 
tions; role-playing their response; and drawing, 
using graphic organizers, or other visual aids. 

5. Performance assessment in the language arts 
should include some tasks that students do on their 
own and some tasks that require cooperative efforts. 

6. Performance assessment should be part of long- 
term instruction, not something that happens once 
in a while. In order to accomplish this integration 
in a constructive manner both assessment tasks and 
instructional tasks must focus on authentic activities. 
District level scope and sequence guidelines for the 
language arts should be updated to reflect insights 
from current research and the requirements of 
KERA. 

7. A tremendous amount of training and support 
needs to be provided to Kentucky's teachers. The 
desired impro\' -ments in assessment and instruction 
envisioned by KERA will take more time and effort 
from teachers and administrators who are struggling 
to meet current demands. For example, if students 
are expected to engage in more writing, then 
teachers will need additional time and training so 
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they can respond in ways that help their students 
grow as writers. In addition, teachers and ad- 
ministrators will need additional assistance in help- 
ing students, parents, and the community 
understand the changes that are made in how we 
assess growth in the language arts. In addition to 
these points, the discussion groups thought the fol- 
lowing examples were useful in illustrating work- 
able performance assessment tasks in the language 
arts. These performance assessment tasks foais on 
reading and writing for understanding, and reading 
for pleasure. These are suggestions for students in 
grade 4; the tasks would be adjusted for younger or 
older students. 

Language Arts Goal: Reads to understand a 
variety of print materials for a variety of purposes. 
Writes to communicate ideas to a variety of audien- 
ces. 

Performance Assessment Task 

Writing in Response to Reading. The student will 
read two different selections on a common topic. 
Possible examples might include two different 
editorials dealing with the same issue (e.g., crime 
control, pollution legislation); two different authors 
describing the same event (e.g., growing up in the 
South); or two different authors describing historical 
events (e.g., a Northern and Southern view of the 
Civil War). 

The student will then write a revSponse that analyzes 
and evaluates the two different reading selections. 
In addition the student is able to communicate his 
or her own reaction to, or view of the issue or topic 
under consideration. 

Potential Questions could include the 
following: 

1. Which author presents a more convincing argu- 
ment? Why? 

2. Could the arguments presented by both authors 
be valid? If so, how? If not, why not? 

3. How is your view of the topic different from or 
similar to the views presented by these two authors? 

4. How do you think these two authors would 
respond to your interpretation or point of view? 

Performance Criteria 

Possible Performance Criteria could include the 
following: 



1 . Comprehension - the student is able to under- 
stand the gist of the selections he or she read. 

2. Analytical/Critical Stance - the student is able 
to compare and contrast the selections in terms of 
the quality of the arguments. 

3. Personal Response - the student is able to 
articulate his or her own points of view and compare 
and contrast this view with those expressed by the 
two different authors. 

Scoring Rubric 

Rxpert Perfnimance 

The Student displays an excellent understanding of 
both selections. He or she is able to compare and 
contrast the writing of the two different authors, and 
he or she is able to develop and defend his or her 
own interpretation. 

Acceptable Performancft 

The student displays some understanding of both 
articles. He or she is able to compare and contrast 
the two selections, but only along a relatively few 
dimensions. The student's personal interpretation is 
relatively short and without much substance. 

Sub-standard Pftrfnrmancft 

The student displays little understanding of either 
article. His or her own interpretation of the issue or 
topic under consideration is weak. 

Language Arts Goal: Reading is a major source 
of personal enjoyment, inspiration, and growth. 

Performance Assessment Task 

1. Ask the student to bring up a book selected from 
his or her list of books and magazines read for 
pleasure (e.g. during free reading at school). 

2. Have the student talk about the book. Possible 
questions include: a. Why did you pick this book to 
read? b. Tell me about this book? c. Would you 
recommend this book to a friend? 

3. Ask the student about others books on his or her 
list. Possible questions include: a. Which authors do 
you like the best? b» Why do you hke their books? 
c. What kinds of things do you like to read about? 

Performance Criteria 

Several criteria could be applied to this perfor- 
mance task, but three of the most important follow: 

1. Enthusiasm or other evidence that the student 
views reading as relevant to his or her personal life. 
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2. Evidence that the student understood the gist of 
the book or experienced a strong personal response 
to something he or she read. 

3. Evidence that the student engages in reading for 
pleasure on a regular basis. 

Scoring Rubric 

Expert Pp -rfnrmance 

The student has an extensive list of books and 
magazines read for pleasure. The student's descrip- 
tion of the book reveals an appreciation of the 
author's style. The student reveals some insightful 
understanding of how the book is relevant to his or 
her own life. 

Acceptab le Pftrfnrmance 

The student*s list of books is relatively short, 
perhaps the student is reading only short, familiar 
books. The student knows a limited number of 
authors. The student is able to remember some 
things about the book, but exhibits relatively litde 
endiusiasm or understanding. 

Suh-standarH Pftrfnrmance 

The student's list of books read for pleasure is 
extremely short or non-existent. The student has 
difficulty idemifying favorite stories or authors. 
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he following points were made/discussed at 
the Practical Arts session: 



1. The goals, philosophy and traditions of the 
Practical Arts lend themselves more readily to per- 
formance assessment than do some other subject 
areas. 

2. The Assessment Component of the Kentucky 
Competency Based Vocational Education (CBVE) 
Curriculum is performance based. CBVE is based 
on validated tasks which are specified in measurable 
behavioral objectives. 

3. Physical Best is an illustration of an effective 
assessment based program. Each student identifies 
his/her physical limitations based on national stand- 
ards and then sets goals. Periodically students 
assess progress toward accomplishinj the goals, 
establishing new goals as appropriate, 

4. Performance assessment will be labor inten- 
sive. In the Practical Arts it will also be expensive 
in terms of products and equipment. 

5. A problem in assessing students in any subject 
area is that they bring their deficits in other subjects 
to the subject under review. 

6. Performance based assessment instruction 
might be organized by learning tasks that require 
interdisciplinary instruction. Secondary teachers 
will find interdisciplinary teaching more challenging 
than will elementary teachers. 

7. The Skyline Vocational Center in Dallas, Texas 
has an impressive system of keeping records on 
student performance. 

8. The Kentucky Education Reform Act specified 
that all students will be successful. 

• It will be difficult to define success, 
since what is successful in one person is 
not necessarily considered successful in 
someone else. 



• Since a person's ability to make a living 
is often considered a sign of success, 
business and industry ought to have 
input into the definition of success. 

• In order for students to be successful, 
they will need to be guided into classes 
compatible with their needs, aptitudes, 
and interests. 

• Students could be motivated to be suc- 
cessful by incentive programs in which 
they earn items (e.g., badges) based on 
level of proficiency in learning task com- 
pletion. Youth organizations which 
have proficiency incentive programs in- 
clude Boy Scouts, Girl Scouts, and 
various vocational education student or- 
ganizations.* 

9. KERA requires measurement of the proportion 
of students making a successful transition to work, 
postsecondary education, and the military. Criteria 
to be used in measuring success is of concern as is 
the manner in which data will be collected. 

10. Training via satellite transmission may be an 
efficient way to prepare educators to do perfor- 
mance assessment. 

11. A new position that may evolve in local 
districts is assessment specialist. University 
programs may need to be developed to prepare 
people for these jobs. 

12. School based decision making groups will 
need to be well versed in performance assessment. 
Further, they will need good data on which to make 
decisions. It will be incumbent on districts to pro- 
vide the best data possible in order for groups to 
make the most informed decisions. 

13. In order to develop a truly effective perfor- 
mance assessment system for Kentucky, its purposes 
will have to be decided and clearly stated. 

♦Vocational education was assumed to be a Practical 
Art since no other workshop at the Conference encom- 
passed it. 
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Arts and Humanities 



Ronald Schmelzer 



George Szekely 

As defined in our discussion, pertormance 
based assessment is the student demonstrat- 
ing his or her skills, concept attainment 
and/or creativity in the arts or humanities. These 
skills, ideas that have been learned and creative 
performance, may be evaluated against three criteria 
that are relative to: peers, personal growth, and a 
predetermined objective. A student*s performance 
may be judged as Basic (having achieved minimal 
competency); Proficient; and Advanced. 

Participants agreed that in the performing arts, 
e.g., music, visual arts, dance and drama, emphasis 
should be placed on the creative aspects of the art. 
This is not to say that skills and concept development 
should be slighted; although in some instances, this 
may legitimately occur. 

It was generally agreed that more questions had 
been raised than answers given. Some of the ques- 
tions raised follow: 

1. How much responsibility and freedom does 
the individual teacher have in developing his or her 
own criteria for performance? 

2. What role does the students* past experience 
in the arts have in performance assessment? In other 
words, should a student who has a good deal of 
competence be judged by the same criteria as one 
having no competence? 

3. Is there a common cannon for the arts? If so, 
what role will it have in creating rubrics for assess- 
ment? 

4. What role should professional associations 
have in developing performance objectives? 

5. Should the same criteria be applied to different 
regions of the state? ^^ifferent subcultures? 



6. What effect does the quality of instruction have 
on the attainment of performance standards? If 
performance is influenced by instruction should all 
teachers be equally competent? How will this be 
assured? 

In the afternoon session there was much discussion 
and Uttle agreement. Some of the comments and 
questions follows: 

1. What is performance based assessment? 

2. Performance based assessment has been ill- 
defined. 

3. Little to no guidance has been given as to how 
to implement. 

4. Teachers are once again being given the 
responsibility to implement something, but not the 
freedom or authority. 

5. An overall structure needs to be developed 
and communicated that will allow for the develop- 
ment of a reasonable performance based assessment 
plan. 

6. PoUticians and administrators need to give the 
teachers time outside of the classroom to develop 
performance based assessment. 

7. A consensus needs to be developed in regard 
to the value of performance based assessment. 

8. Many, if not most, administrators fear that 
their authority will be usurped by site based manage- 
ment and performance based assessment. A conse- 
quence of this is that it is being "bad-mouthed." 

9. Staff development is critical to the develop- 
ment of performance based assessment. 

10. Teachers need to be provided an environment 
that is conducive to collaboration, including com- 
mon planning time. Unless this happens, there will 
a need for additional sUff; indeed, this may be the 
case anyway. 

1 1 . Constraints place upon a teacher's instruction- 
al time must be hfted. 

12. Colleges and universities must take the lead 
in guiding school districts in the development and 
implementation of performance based assessment. 
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Group Leaders' Comments: 

The group experienced a great deal of anger and 
frustration during the session. This may have oc- 
curred because few, if any, of the participants had 
an understanding of what performance bas^ assess- 
ment is, much less how to implement it. 

We believe that there is a critical need for staff 
development on a state-wide basis in order to clear 
up misimderstanding and to help teachers and ad- 
ministrators recognize the value of performance 
based assessment including the critical need for 
site-based management as integral to its implemen- 
tation. 
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Virginia Davis Nordin 

Joseph M. Petrosko 

The Kentucky Educational Reform Act 
(KERA) is a vaguely written law. Com- 
pared to other educational laws, it has many 
undefined terms and many areas of ambiguity. The 
intentional vagueness of KERA presents both 
problems and opportunities. One problem might be 
an excessive concern for performance outcomes. 
Schooling might become totally redirected to the 
achievement of observable outcomes and nothing 
else. However, the vagueness of KERA provides 
opportimities to educators. For example, there may 
be a heightened awareness of the needs of deprived 
students. Issues of economic equity will become 
obvious when both rich and poor students start being 
held accountable to the same performance stand- 
ards. 

A problem with performance assessment (as it has 
evolved so far) is lack of local ownership of the 
process. Teachers and school personnel do not have 
a feeling of participation. Various task forces of the 
Council on School Performance Standards, for ex- 
ample, task forces in mathematics and language arts, 
do involve substantial numbers of teachers. How- 
ever, the great majority of Kentucky teachers are 
merely onlookers to the process of writing perfor- 
mance assessment tasks. Better communication 
links to the local level are needed. 

Comparability and accountability are key issues in 
the Kentucky school reform. Each school will be 
held accountable for the achievement of its students, 
but how fair is this when schools are not com- 
parable? The law recognizes that not all schools are 
the same. Nevertheless, the manner in which dif- 
ferent schools will be treated is presently unknown. 

Local control of schools, in the form of site-based 
management, is relevant to the discussion about 
performance assessment. According to the law, 



each local school council will adopt a policy regard- 
ing "determination of curriculum. . .curriculum 
development. . .[and] alignment with state stand- 
ards. " Does this mean that local councils set goals 
for achievement? If so, will there be a procedure 
to assure that there is comparabihty among schools 
in a district? Furthermore, what about goals coming 
from the state department of education? Model 
curricula for elementary and secondary education 
are supposed to be published in 1993. But during 
1991-1992, interim tef ig will have already oc- 
curred to establish a baseline for assessment 
decisions to be made during the 1993-1994 school 
year. So what is the point of the state's model 
curricula? It seems that schools are held responsible 
in the assessment procedure, but with little control 
over or understanding of standards. 

The method of reporting results of performance 
testing will be very important. State assessment is 
designed to focus on the individual school. How- 
ever, the news media are sure to take the published 
data from state assessment and attempt to rank 
districts, just as was done with the Kentucky Es- 
sentials Skills Test (KEST) and Comprehensive 
Tests of Basic Skills (CTBS). This will probably 
have a negative effect on some districts. How can 
this be avoided? 

The relevance of performance testing to parents is 
questionable. Many parents are solely concerned 
with how their child performs and compares with 
others. They are not concerned with how the school 
performs. Results of traditional testing— the con- 
tinuous assessment program at the district level- 
may be of more interest to parents than results of 
performance testing. 

The exact relationship between performance test- 
ing and continuous assessment program testing has 
not been worked out. For example, it is conceivable 
that students in a school could do poorly on perfor- 
mance testing, but do well on a standardized test 
used in its district's continuous assessment program. 
Such a discrepancy could undermine the credibility 
of rewards or sanctions given to the school. 
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According to KERA, schools will receive rewards 
or sanctions depending on changes in the percentage 
of successful students. The precise meaning of 
successful gtudent is currently unknown, and will 
require a lot of effort to specify. The state depart- 
ment of education has produced a table, a "Reward 
Matrix," which lists the consequences of each 
outcome (e.g., what happens when the percentage 
of successful students decreases by more than 5%). 
Most educators believe that this has an air of un- 
reality. There are too many unknowns. Too many 
things have to transpire between now and the 
decision dates shown on the matrix to take these 
various consequences seriously. 

Someone should be taking note of the unintended 
consequences of KERA, For example, in im- 
plementing performance assessment, we may end 
up with students who perform competently during a 
testing situation, but who never develop the disposi- 
tion to perform well on their own. They will be 
good test takers, but not very good life-long 
learners. 

Assessment tasks need to be relevant to diverse 
populations. An example of an assessment task 
proposed for use in the schools requires students to 
prepare an oral report on the likelihood of an 
earthquake in Kentucky. The purpose of the report 
is to help the mother or father of a student to decide 
on the purchase of earthquake insurance. In some 
schools, many students do not live with either 
parent. Also, students do not live in a house, but 
rather an apartment or housing project. Finally, 
many students do not know what insurance is be- 
cause their families do not have it. 

Afternoon session 

Educators should learn about the concept of tech- 
nical amendment to a law. The KERA of 1990 is 
sure to require such amendments. They could be 
put into a bill that would be voted on by the General 
Assembly. Technical amendments would clarify 
and modify ambiguous portions of the law. Tney 
would not change the essential nature of the law, 
only refine it. Educators should think about propos- 
ing specific, practical technical amendments. 

A number of things in KERA could stand modifica- 
tion. For example, the law requires performance 
assessment at grades 4,8, and 12. It might make 
more sense to begin testing at grade 3 rather than 
grade 4 (especially in Hght of the shift to ungraded 



early primary school). Further, the relationship 
between assessments required under Sections 2 and 
3 of KERA needs clarifydng, as well as the relation- 
ship between curriculum and assessment respon- 
sibility. 

Although the vagueness of the law allows 
educators flexibility, the strong sanctions for 
"failure" (the latter essentially undefined) seems 
unfair and seems to lead to unnecessary and non- 
productive anxieties. 

At the present time there are no legal requirements 
in effect except those for continuous testing. The 
requirements will arise only after the Council on 
School Performance Standards and other groups 
issue various standards, guidelines, and regulations. 

The costs of implementing performance assess- 
ment will be substantial. A lot of time and money 
wiU be needed to train teachers. There wiU also 
need to be a system for sending out material from 
one school district to another. Biases could occur 
if teachers from a school tried to assess their own 
students. 
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Implementation and Management 



Kawanna Simpson 
ScGtt Trimble 



Ken Draut 



Tom Mowery 



The puipose of the session was to give school 
leaders a better idea of their responsibilities 
to manage the new testing program. The 
process was for Draut and Trimble to present im- 
portant information for 30 minutes followed by 
group discussion of several issues. 

What has student assessment in 
Kentucky looked like in the past? 

Trimble gave a historical sketch of Kentucky man- 
dated testing program: 

• mid 1960's the state began helping to 
score state tests to be used for standard 
comparisons on a voluntary basis. 

• early 1970's a need grew for regional 
and state testing data. 

• 1978 the state testing program changed 
focus with the Education Improvement 
Act that required testing in grades 3, 5, 
7 and 10. 

• 1984 the KEST was mandated that in- 
creased the level of accountability— also 
provided estimated norms. 

• 1988 the legislature mandated a norm- 
referenced test with no estimated norms. 

• brings us to 1990 KERA and a whole 
new ♦'^sting program. 



What went wrong with past 
assessment programs? 

The problems in Kentucky were not unlike national 
problems: 

• Due to rankings of school districts, im- 
mense pressure was placed on norm-ref- 
erenced tests. 

• The samples of curriculum on the tests 
frequently became the curriculum-too 
much focus on items rather than the 
whole curriculum. 

• The curriculum became too narrow, 
which tampered with the validity of the 
tests. 

• The norms could not be updated annual- 
ly, leading to most states placing above 
the national mean. 

What is performance assessment? 

Draut led a discussion of the session materials 
which included samples of performance assessment 
from other states and from Kentucky sources. Many 
teachers have for years routinely included similar 
activities in their curriculums. 

There will be pressure for teachers to change the 
way they teach so that methods reflect the new ways 
of assessing student performance. 

How are performance assessment 
items scored? 

The scoring of performance based assessment 
items will generally be more complex than past state 
testing programs. 

Much of the scoring will occur at the school level 
rather than sending answer sheets to Frankfort for 
scanning and scoring. 
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Teachers and others must be trained to judge 
student performance in ways foreign to many. As 
examples of scoring difficulty, OM and Governors 
Cup Future Problem Solving competitions were 
noted. Many educators are aware of the difficulty 
in training officials for these competitions, and the 
large intervals of time required to judge them. 

Question: Who will train liie judges? When? 
Where? Cost? 

Training must be a joint effort of higher education, 
Department of Education, local schools and others. 
Most of the cost will probably be paid by local 
school districts. KET must nave a key place in the 
training of judges. 

Question: Will additional in-service days be 
provided for teachers and administrators to be- 
come trained? 

This is a concern that must be addressed by the 
legislature, but training is a priority. 

Question: Who will be trained to write the assess- 
ment items that frequently require much atten- 
tion to detail so that students know exactly what 
is required of them? 

Question: Proper judging (i,e,, reli^ihle judging) 
is essential to prevent lawsuits since sanctions and 
rewards will be based on the outcomes. Are we 
underestimating the training of judges? 

Question: Isn't the only true way to have reliability 
is to use the same items over and over across all 
school buildings? 

What are some major issues in the 
state performance assessment program? 

• Sampling, reliability and validity are the 
three issues. 

• Design of the test will have to take these 
issues into account* 

What are some examples of current 
statewide assessment programs? 

Draut addressed what some other states are doing. 
Arizona in particular has developed an interesting 
system. They develop parallel tests, one to be given 



for teacher use early in the school year, and the 
second to be given in the spring semester for assess- 
ment purposes. In addition, Arizona gives a multi- 
ple choice norm-referenced test each fall semester 
to meet the demands of diose who want such infor- 
mation. 

How will performiince assessment 
impact local school assessment? 

• Local districts must have a plan for con- 
tinuous assessment that goes beyond the 
state program. It seems reasonable that 
schools will continue to give traditional 
tests to meet local information needs, 
such as determining basic skill achieve- 
ment, communicate with the com- 
munity, and to have Chapter I data. 

• Schools should also give tests similar to 
the state assessment to students in 
grades other than the three grades man- 
dated by the state program. 

• To have a quality local program, dis- 
tricts need to provide staff training using 
writing prompts and other prompts. 

Question: What should a school do first? 

Vocational and special education teachers have 
been using performance assessment for some time 
to measure outcomes. They might be a good first 
source for ideas. 

Be thinking about how technology can be used to 
help with performance assessment, both in the as- 
sessment phase and the judging phase. 

Question: Who determines the mastery level of a 
task? 

Until the state gives some indication, schools 
should be experimenting with mastery levels. 

Question: Will the Department score CTBS tests 
this spring? 

They will score them this year and perhaps every 
year, depending upon the new commissioner and 
new state board. 

Question: What can schools be doing in the next 
six months? 

There are only best guesses right now about the 
new testing program. All schools should begin 
experimenting and practicing. Some national 
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groups are working with mastery learning, so 
teachers should get active in their national and state 
associations. Get materials from OM and Future 
Problem Solving competitions and scouting merit 
badge books. These sources of performances are 
handy even though rubrics are not given in all cases 
to judge the performances. 

Question: What should schools do to prepare 
students for ACT and SAT tests for college 
entrance? 

Scores on these tests will continue to be a measiire 
of school program success. Schools should con- 
tinue to have multiple choice tests as a part of their 
continuous testing program. 

Question: Is the Department of Education still 
going to oJ Ter the statewide in-service on student 
performance assessment on February 15? 
Yes. 

Question: With all the demands of KERA on local 
school staff, how can they find the time to train 
in all the areas? 

This is a concern that must be addressed quickly 
by by proper state authorities, but it could be that 
local school officials will have to find a way for 
training at the local level. 

Question: How will the School Performance 
Standards Council impact the performance as- 
sessment program? 

There is a close working relationship between the 
two groups. 

Question: Since the state did not pay $300 for 
positiye teacher evaluations under a 1985 state 
program, how do teachers know that money will 
be available for rewards and also to provide 
service to schools in crisis? 

There is never any certainty that a program will 
be entirely funded^ but the tax changes and legisla- 
tive commitment seem to indicate funds will be 
available. 

Question: How will the state inhibit cheating in 
the new program? 

A new code of ethics is needed. 

It could be that there will be testing centers inde- 
pendent of the Department or schools to which 
students go to be tested. Or> teams of educators 



from one district could go to another district to 
administer and judge the performance assessment. 
Outside monitors or administrators may be used. 

What are the barriers. If any, to 
implementing performance 
assessment? materials? costs? 
training? 

The costs of the program have probably been 
underestimated, the materials will have to be 
developed, and the training will be cosdy in dollars 
and time. 

How will school based decision making 
and performance assessment impact 
each other? 

• Teachers at each school will have 
rewards and sanctions impacting their 
Uves due to performance assessment. 
They have, then, a vested interest in 
how their students perform. 

• Many teachers will want more say in 
the treatment of students since their 
careers might be at stake. 

Discussion questions: 

1. How will the following elements of reform 
impact performance assessment: 

• Accoimtability (rewards and sanctions) 

• Ungraded Primary 

• Special Education 

• Administrative and Instructional Technol- 
ogy 

• Preschool Programs 

2. What management problems can be predicted 
when implementing performance assessment in 
elementary, middle and high schools? 

3. How can the state pay for rewards for schools 
and also provide service to schools in crisis? 

4. What changes in information needs will be 
required due to performance assessment? 
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5. What type of materials should schools be 
developing in preparation for performance assess- 
ment? 

Additional questions from the session 
participants: 

Question: Will the Department provide guidelines 
for Chapter I testing? 

Yes, but these generally come from federal man- 
dates. 

Question: What areas will be assessed? 

At least the six goal areas listed in the Reform Act 
will be tested. 

Question: Should teachers teach the test? 

It appears that the intent of performance assess- 
ment is to change the way teachers teach. It is 
hoped that the way teachers teach will become 
natural preparation for the tests. In fact, teachers 
should use perfornance assessment routinely to 
grade students. Teaching to the test will be a 
positive experience for teachers and students. 

Question: How can we prevent the Lexington 
Herald Leader newspaper from ranking schools? 
Cannot. In fact, schools may start being ranked. 

Question: How is a school's threshold changed 
when a drastic population shift occurs that might 
significantly increase or lower the threshold? 

A formula must be developed to account for such 
shifts. 

Question: How will this program provide national 
comparisons for Kentucky schools? 

And some other Questions: 

Question: How will this program make better 
teachers? 

Question: If we have not been successful in teach- 
ing lower order thinking skills, what makes 
anyone think that we'll do better with higher 
order thinking skills? 

Question: Why were 12th graders chosen to take 
the exam, given their likely uncaring attitudes? 



Question: When will colleges catch up to perfor- 
mance assessment? 

Question: Where is student accountability in this 
assessment program? 

Question: How will public schools compete with 
private schools given this experimental program? 

Question: Since multiple choice tests take less 
time, are objective, are easy to score, and many 
questions can be covered, what are the ad- 
vantages of performance assessment? 

Question: What if a school threshold is so close to 
the top of possibilities that they can't easily get 
rewards? 

Question: When must a LEA have a continuous 
assessment program in place? 

Question: Must a school district include kinder- 
garten in their local assessment program? 

Question: Will the Department continue its test 
item bank? When will it have performance as- 
sessment items in it? 

Question: How will performance assessment help 
determine if a child is ready to graduate to the 
fourth grade? 
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George Cunningham 

Skip Kifer 

Matrix or item sampling is a set of meas- 
urement techniques and linked statistical 
methodologies which under certain cir- 
cumstances allows one to save time, money (or both) 
and increase the amount of information gathered 
when implementing a testing program. These pro- 
cedures are used extensively in the National Assess- 
ment of Educational Program (NAEP) and 
state-wide assessment programs. 

Consider, for instance, the problem of estimating 
the percent of twelfth graders in the Commonwealth 
who can respond at a satisfactory level to the 
question: 

Why in the United States are slave insurrections 
called revolts, not wars? 

One approach to estimating the proportion of 
students who answered at a satisfactory level would 
be to give the question to all students in grade 12 
and then calculate the proportion who gave satisfac- 
tory answers. That would mean measuring 
thousands of students in about 300 high schools. 
This method produces precise, valid, and useable 
results, but is not efficient in a statistical sense. 
Comparable results could be gotten from responses 
of many fewer students. (The Gallup poll estimates 
percents of the U.S. public on sample sizes around 
1000.) 

Approaches using item or matrix sampling could 
change the above design in two major ways. First, 
because there are so many 12th grade students, one 
could define four tasks (instead of one) and random- 
ly assign one those tasks to each student within each 
of the schools. One could now estimate four propor- 
tions (albeit with slightly less precision) for about 
the same amount of time and money using the same 
population that provides just one estimate in the first 
scenario. Hence, through the <;nmpling oF tasks. 



one has increased the amount of information 
gathered per unit of time and money. 

Second, one could take a sample of schools and 
administer just one task to all twelfth graders in the 
school. That sample would be used to estimate the' 
proportion of students within the Commonwealth 
who gave satisfactory answers to the question. 
Hence, through the <sampling of units, one has 
saved money (and perhaps time) but still obtained 
the desired estimate. 

A combination of the two major ways of sampling 
r imits and ta sk^;; nr matnv samplin^^ provides a 
more efficient way still to gather information. This 
time one could sample schools and give samples of 
tasks. For example, one might define eight tasks 
and give a different subset of the eight to a sample 
of schools. In this scenario one is sampliu ; both 
tasks and schools and is, theoretically saving money 
(perhaps time) and getting increased information 
about a broader array of tasks. This kind of sam- 
pling can be expanded so long as there are large 
enough samples of students taking each task to 
estimate precisely a proportion of interest. 

The notions of matrix or item sampling figure 
prominently in the language KERA. No distinction 
is made, however, among task, unit or matrix 
sampling. It is clear that a census fthe first example 
where all students do all things) of students and 
tasks is not required. The language does, however, 
allow for the possibility of task sampling, unit 
sampling and both (matrix sampling). To the extent 
the law requires a state-wide assessment, a form of 
sampling is a way to get desired information for less 
cost and time than by having all students respond to 
all questions. In this sense item sampling would be 
used in the Conmionwealth just as it has been used 
in NAEP or other state-wide assessments. 

A second use of item sampling was apparently 
envisaged by those who framed the law. The ac- 
countability portion of the legislation makes a school 
the unit which could receive rewards or sanctions. 
Item sampling, theoretically, would allow one to 
estimate a percent of successfiU students in a school 
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without requiring all students to do all tasks. Hence, 
item sampling is viewed as a cost effective way to 
implement the accountability arm of the legislation. 
It is in this second use that the intent of the 
legislation may be pushing the psychometric limits 
of item sampling technology. Three features of the 
law or Kentucky's schools should be noted in this 
regard: 

• There are small schools in Kentucky and 
there are schools with substantial tran- 
sient populations of students. 

• Comparisons to be made are between 
cohorts, Tnat is, performance in 1992 
for grade 4 students is to be compared to 
performance of students in grade 4 in 
1994. 

• The precision with which one must es- 
timate the percent of successful students 
is less than one percent. That is, a 
school that is to receive a reward must 
do one percent better than its threshold 
value. 

These features, individually and in combination, 
place sever<i pressure on this set of methodologies. 
The premis t of item sampling is a large population 
of things in this case students) which can be 
sampled v ithout a great loss of precision when 
making st tistical estimates. Suppose, however, 
that a scb x)l had just fifty 4th graders. If each 
student w jre to do one task, a response would be 
"worth" r ffo percent. If each student were to do one 
of two tf sks, the response would now be worth four 
percent. The implication is clear. Each student 
would x\ft^<l to do a lot of tasks in order to estimate 
with the desired precision. 

Suppose two years latter there was a marginal 
change in the cohort. This new class had two 
students in it who could do everything well while 
the class that formed the baseline had only one such 
student. That one additional successful student 
could have a profound effect on the estimate of the 
percentage of successful students, the comparison 
between levels of that percent, and whether or not 
a school received a reward. 

This example is meant to be just that. The 
problems of estimating the percent of successful 
students is difficult regardless of the size of school 
and compounded with cohort comparisons and the 



need for extremely precise estimates of a school's 
performance. 

It may be the case that matrix sampling will be of 
limited utility in the accountability portion of 
KERA. A census of students with some tasks (each 
student does a number of common tasks) or task 
sampling may be more appropriate. 
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Performance Assessment for Special 
Populations 



Deborah Bott Slaton 

Jay McLaughlin 

Teachers, administrators, and teacher 
educators attending the conference on per- 
formance assessment and the session deal- 
ing with special populations expressed several 
concerns regarding how students identified as hand- 
icapped or eligible for Chapter I services would take 
part in the performance assessment program 
developed for the Kentucky educational Reform 
Act. Participants also noted some similarities be- 
tween procedures for performance assessments and 
some informal assessment techniques used to 
monitor progress of some students with special 
needs. Finally, participants discussed existing as- 
sess ment requirements for federally funded 
programs such as Chapter I and programs for stu- 
dents with identified disabilities. 

Concerns About Participation 

Session participants expressed concerns regarding 
how students with disabilities would be included or 
excluded from a state-wide performance assessment 
program. In some instances, the educational 
programs of students in special education will not 
match the content of assessment instruments 
designed for the non-handicapped school popula- 
tion. For example, students with moderate or 
severe mental handicaps would typically be working 
in a program that emphasizes fiinctional, daily living 
skills rather than traditional academics. In other 
instances, specifically students with mild disabilities 
who spend only a portion of the school day away 
from the regular classroom, the content of educa- 
tional programs of students with disabilities may 
match that of the non-handicapped school popula- 
tion although the match may not be with age-mates. 
Including special education students who spend time 
in the regular class in the performance assessment 
program could increase classroom teachers' sense 
of accountability for the rate of learning among 



students with disabilities. This was viewed as a 
potentially positive outcome of performance assess- 
ment 

Participants did not recommend that all studwxits 
identified as handicapped be exempted from the 
performance assessment program. Rather, the con- 
sensus was that the Individualized Educational Plans 
(lEPs) required by federal regulation could be used 
to determine participation/exemption in perfor- 
mance assessment activities. This option would 
allow educators and parents to make decisions 
regarding participation on an individual basis, but 
teams would need guidelines and criteria for how to 
approach this decision. One possible component of 
the decision could be the amount of overlap found 
between curricula in the mainstream and the 
student's lEP goals and objectives. 

Participants voiced concerns about how the 
rewards and sanctions tied to outcomes of perfor- 
mance assessment could pressure some schools with 
high numbers of students with special needs to 
automatically exclude these students from any per- 
formance assessment program. There were also 
questions regarding how special day schools and 
residential schools for students with disabilities 
(e.g., Kentucky School for the Deaf) would fit into 
the performance assessment prof^.am. Careful sam- 
pling procedures could control that the proportion 
of students with special needs present within testing 
sample is representative of the school or district 
student population. 

Existing Assessment Practices in 
Special Programs 

In some instances, existing assessment practices 
within special education programs may parallel or 
exempUfy performance based assessments. For ex- 
ample, assessments of daily living skills using an 
ecological approach and community catalogs often 
require students to perform real tasks in community 
settings (e.g., using a city transportation system, 
shopping for groceries, etc.). Some of the cur- 
riculum-based assessment techniques used to assess 
lEP objectives may also be similar to performance- 
based assessments, depending on the nature of tb^^ 
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EBP objective. It may be possible for some assess- 
ments already conducted by special educators to be 
supplementary to and supportive of the performance 
assessment program. These examples of 
similarities among assessment procedures will not 
eliminate the need to include teachers working with 
special populations in staff development activities 
related to performance-based assessment techni- 
ques. 

Existing Assessment Requirements in 
Programs Receiving Federal Funding 

Several of the federally funded programs for stu- 
dents with special needs (i.e., Special Education and 
Chapter I) must comply with federal regulations 
regarding assessment, and the performance assess- 
ment program may not be an acceptable substitution 
for existing practices. Examples include the follow- 
ing: 

• assessments required to establish 
presence of a disability and eligibility 
for special education 

• ongoing monitoring and annual review 
of IBP goals for each student placed in 
special education 

• screening of all primary students for 
eligibility for Chapter I services 

• d) spring-to-spring administrations of 
norm-referenced tests required for stu- 
dents placed in Chapter I. 

Participants did not foresee that the performance 
assessment program established under KERA will 
satisfy all program needs for data to modify instruc- 
tion, inform parents, or identify which students are 
eligible for special services. Participants also raised 
concerns regarding confidentiality requirements for 
records of students identified as handicapped and the 
potential logistical problems involved if several 
different layers of assessment are required by 
federal and state programs. 

Federal mandates for nondiscriminatory testing 
and reasonable accommodation must be part of any 
performance-based assessment program that invol- 
ves individuals with identified disabilities. Again, 
the team writing the lEP is probably best suited to 
specify what accommodations an individual student 
would need to participate in a performance assess- 
ment program. Examples of testing accommoda- 
tions include extra time, braille writers, and use of 
audio tapes to substitute for printed materials. The 



concept of "reasonable accommodations'* should be 
used to guarantee that, when appropriate, students 
with disabilities participate fully in the performance 
assessment program. 

Summary 

There was a general consensus among session 
participants that students with special needs were 
not considered during the development of the Ken- 
tucky Educational Reform Act. The task before us 
now is to develop clear statements regarding how 
students served by special programs will take part 
in all aspects of KERA including performance as- 
sessment programs. 
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Paul de Mesquita 

Malessa Billings 

This paper summarizes issues, viewpoints, 
and questions of a sample of Kentucky 
public school pupil services personnel who 
participated in group discussions focusing on the 
implementation of the Performance-Based Assess- 
ment section of the Kentucky Education Reform Act 
(K.E.R.A.). The group discussions were conducted 
as part of a series of workshops on performance 
assessment held at the University of Kentucky, 
January 4, 1991. A total of 26 discussants attended, 
representing superintendents, supervisors, prin- 
cipals, counselors, parents, school board members, 
and university faculty. Throughout the discussion, 
many of the issues raised revolved around four 
central themes: background and definitional ques- 
tions, roles and responsibilities, implementation 
concerns, and future direction. This sunmiary has 
been organized accordingly to reflect these four 
major themes. 

1 . Background & Definition 

What is the purpose of performance-based assess- 
ment? Before a meanirgful discussion of perfor- 
mance-based assessment could occur, some review 
of the background and context surrounding these 
issues was required. The state-wide educational 
reform plan for performance-based assessment is 
still relatively new and unfamiliar to most of 
Kentucky's educators. Consequently, many 
educators misunderstand the intent of such an as- 
sessment effort. The primary purpose of estab- 
lishing a performance-based assessment program 
was to "insure school accountability for student 
achievement" of educational goals outlined in the 
reform act. A secondary purpose of the assessment 
program is to present a basis for comparing the 
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performance of Kentucky students with students 
from other states across the nation. 

What will be assessed? Beyond the goals and 
stated purposes of performance-bases assessment, 
another area of misunderstanding involves the focus 
of the assessment. The Kentucky Education Reform 
Act clearly delineates seven capacities that serve as 
the educational outcomes for the Commonwealth's 
system of public education. These broad areas 
include comimunication skills, knowledge and un- 
derstanding of political and governmental proces- 
ses, mental and physical well-being, cultural and 
historical appreciation, work skills, and skills to 
compete academically. The seven broad capacity 
areas will be translated into measurable goals and 
student outcomes that will serve as the basis for 
assessment of students* performance. Specific stu- 
dent outcomes will include not only achievement in 
traditional core subjects, but emphasize self-suf- 
ficiency, personal responsibility, thinking and prob- 
lem-solving skills. Given the legislative purpose of 
performance-bases assessment and the stated out- 
comes that are to be assessed it is clear that perfor- 
mance-based assessment will have little 
resemblance to the more traditional standardized 
educational achievement test reUed upon so heavily 
in the past. 

What performance-based assessment is 
not. 

Fully understanding performance assessment also 
means understanding what it is not. Performance- 
based assessment will not be a standardized testing 
program for all students. Typical paper and pencil, 
multiple-choice questions will not be the 
predominant format. Performance assessments will 
be more open-ended. Instead of filling in answers 
to questions, students may actively create solutions 
to problems. Not all children will participate. Only 
a representative sample of students will participate 
in the assessment process. Performance-based as- 
sessment will not yield the kinds of achievement 
score information that can be shared with parents of 
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individual students, such as a percentile rank. 
Keeping in mind the overall purpose of school 
accountability, the results will be most meaningfiii 
in terms of schools and school systems as a whole 
rather than measures of individual student achieve- 
ment. The relationship between performance-based 
assessment as outlined in K.E.R.A. and annually 
administered standardized achievement testing 
remains unclear at this time. It is possible that these 
two different measurement perspectives eventually 
may be used in a compatible and coordinated way. 
This is an area that pupil services professionals may 
play a contributing role. 

il. Rrie & Responsibilities 

A second major theme of discussion focused upon 
the roles and responsibilities of student service 
personnel. A m^jor area of responsibility probably 
will be that of in. *^.rvice education for school per- 
sonnel. S:^.ve ai p- > jsible areas in which leadership 
could be £.:jumed ^re presented. 

First, public school pupil services personnel must 
bec:)iv"t knowledgeable of why students will be 
assesecx* and what kinds of student performance will 
bp. measured. This awareness and knowledge of the 
inte A and scope of performance -based assessment 
becomes a responsibility for pupil services staff in 
terms of their roie in educating colleagues, teachers, 
and parents about the proposed assessment program. 

Second, since many pupil services staff function 
in positions such as guidance counselor, school 
psychologist, diagnostician, and supervisor, these 
professionals, because of their advanced graduate 
training, should be more knowledgeable about tests 
and measurement concepts. Therefore, they should 
be more informed about curriculum-based perfor- 
mance assessment as a more meaningful measure- 
ment approach. These views can be shared with 
school staff in ways that enhance the acceptability 
of the proposed performance-based assessment 
plan. Ways of integrating performance-based as- 
sessment with continuous classroom assessment 
strategies could be encouraged as a way of monitor- 
ing student progress through the curriculum. 

Finally, many individuals in pupil services posi- 
tions are sensitive to the stresses, frustrations, and 
related problems associated with the anticipated 
transition to educational reform in Kentucky 
schools. The development and implementation of 
performance-based assessment as a statewide school 
accountability system will bring added pressures to 
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educators. Serving as a resource for information 
and guidance concerning perfonnance-based assess- 
ment will be a major role for student services 
professionals. Helping educators imderstand their 
responsibilities and clarify the value of a perfor- 
m^ce-based approach to measuring student learn- 
ing can contribute to maintaining positive attitudes 
and school morale. 

III. Impiementation Concerns 

Throughout the discussion participants identified 
several potential barriers to the implementation of 
performance -based assessment. As alluded to ear- 
lier, lack of information and misunderstandings 
have created an atmosphere of pressure and frustra- 
tion among many Kentucky educators. Relatedly, 
the continuing uncertainty surrounding how the 
assessment program will be developed and imple- 
mented encourages misinformation and increases 
anxiety on the part of schools who will be held 
accountable by such an unknown process. The fact 
that schools will be judged and evaluated by this as 
yet undetermined method has led to wide speculation 
about ways to circumvent the rules through teaching 
to the assessment procedures or overrepresenting 
high ability students in the testing sample. 

Additional questions have arisen as to who will be 
responsible for administering the performance- 
based assessment within a given school or a given 
county. If school or county personnel are involved 
issues of bias and lack of objectivity may arise. It 
is possible that pupil services personnel may in fact 
be assigned to assist with administration or scoring 
the performance assessments. Knowing that un- 
favorable results may jeopardize their school's in 
terms of sanctions and rewards can create enormous 
pressures, not to mention ethical dilemmas. 

From a more practical point of view several areas 
of concern were also voiced. These included how 
much time and effort would be required to ad- 
minister and score the assessments? How much 
time would students He missing from class to par- 
ticipate in this process? At what point during the 
academic year would these assessments be con- 
ducted? What procedures would be used to insure 
that student samples would be random and balanced 
in terms of ethnicity, gender, and socioeconomic 
status. A considerable amount of inservice training 
related to performance-based assessment is neces- 
sary. Finally, what will the costs be in terms of 
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time, effort, and actual test development and will 
these costs be prohibitive? 

iV.Recommendatlons 

Since most of the actual details of the performance- 
based assessment remain ideas and have yet to be 
developed, future directions for pupil services 
professionals are not clear at this time. However, 
the following recommendations are meant to serve 
as a list of practical suggestions for pupil service 
professionals in terms of preparing for the eventual 
implementation of performance-based assessment in 
their school systems: 

1. Begin by studying the Kentucky Education 
Reform Act, Part 1-Curriculum, specifically the 
sections that define and discuss performance-based 
assessment. 

2. Become familiar with performance-based as- 
sessment as an alternative to standardized, pencil 
and paper, multiple -choice format tests. Consult 
with university and college faculty with expertise in 
testing and measurement concepts. Identify 
resource materials on this topic. Begin a school 
resource file with information that can be made 
available or distributed to school staff. 

3. Begin a dialogue with school administrators and 
colleagues about performance-based assessment. 
Attempt to clarify what roles and responsibilities 
pupil service professionals might assume with the 
implementation of the proposed assessment plan. 

4. Assist your schools in identifying the affective 
as well as cognitive outcomes that will be evaluated 
through performance-based assessment as specified 
in K.E.R.A. Encourage teachers to address these 
goals and outcomes through their classroom instruc- 
tion. 

5. Offer instructional consultation to classroom 
teachers on ways to incorporate curriculum-based 
performance evaluations into their classrooms. 

6. Conduct workshops to inform teachers, parents, 
and students about performance-based assessment. 
Clarify misunderstandings. Emphasize the value of 
meaningful assessment. And reduce stress and 
anxiety due to uncertainty and misinformation. 
Focus on the reward aspects of the reform law rather 
than the threat of sanctions. 

7. Establish channels of communication with coun- 
ty and state administrators, as well as legislators, in 
order to provide feedback and offer suggestions 
concerning the development and implementation of 
performance-based assessment. 
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The morning session began with an attempt 
by the group to define perfonnance-based 
assessment. In their definitions, members 
of the group used words and phrases such as: 
hands-on, active, practical, open-ended, subjective, 
the process of getting right answers vs. just the right 
answers, using materials, and not "multiple choice." 
The discussion then focused on the term "authentic." 
The group raised the following questions: Does 
"authentic" involve only consumer-related tasks 
relevant to all citizens in their daily lives? Does 
"authentic" mean the same for average citizens, 
college-bound students, or mathematicians? The 
group decided that mathematics tasks exist which 
are "authentic" for college-bound students and 
mathematicians, but cannot be placed in a consumer- 
oriented situation. In essence, the group suggested 
that many mathematics activities are important for 
their own sake and for developing thinking skills. 

After this discussion, several concerns regarding 
performance-based assessment were raised by group 
members. The first concern was the emphasis on 
reading in mathematics tasks. Some group mem- 
bers were concerned that performance assessments 
would measure reading comprehension rather than 
mathematical expertise. They felt that poor readers 
would also do poorly on mathematics tasks requiring 
extensive reading. 

A second concern was that the length of the tasks 
would cause students to become tired and perform 
poorly. Some members felt that students could not 
concentrate for extended periods of time. 

A third concern was that tasks would be biased and 
prejudicial. Members were concerned that once 
tasks are placed in a specific context, then those 
students with out-of-school experiences in those 
contexts would have an advantage over students with 
limited experiences. They felt that performance 
would be affected by SES and parental involvement 
with their children, rather than what teachers taught 



in the classroom. The group members were shock- 
ed and dismayed by Gary Phillip's comments that 
minority students tended to score more poorly as the 
tasks became more "authentic." 

A forth concern centered on the random sampling 
of students to complete the performance-based as- 
sessment. Group members felt that test scores in a 
district or school co^dd be affected by the particular 
students who are randomly selected to take the 
assessments. Some group members wondered what 
students who were not being assessed would do 
while the randomly selected students were taking the 
tests. The group felt that all students in a school or 
district should be assessed. 

A fifth concern was that only one type of assess- 
ment would be used. Some students perform better 
on multiple choice tests; others perform better on • 
open-ended tests. The group felt that a variety of 
assessments— multiple choice, performance based, 
projects, interviews, etc. -be used to assess mathe- 
matics learning. Some group members felt that 
basic skills should be assessed first and separately 
from performance-based tasks. 

The morning session closed with the two group 
leaders sharing several examples of performance 
based assessment. Each group member received a 
copy of a sample mathematics performance task 
from (1) the Kentucky Mathematics Task Force of 
the Council on School Performance Standards, (2) 
the California Statewide 

Assessment, and (3) the Connecticut State Assess- 
ment. Dr. Bush outlined the assessment procedures 
for college entrance in the Victoria (Australia) Min- 
istry of Education. This assessment consists of four 
parts (projects, problem solving, mechanics, and 
concepts) given at four different times during 
students' final year of pre-coUege schooling. 

Because of the light attendance, this session was 
more conversational and covered many different 
issues. As in the morning session, the vast majority 
of the discussion focused on specific concerns 
regarding the upcoming assessment in Kentucky, 
The first concern was that 12th grade students will 
not take the tests seriously, and, in some cases, 
might try to do poorly on the tests. The group felt 
that eleventh-grade students would be more 
motivated to perform well on the tests. 
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Educators in the afternoon session were con- 
cerned about the length of the assessment. Most 
members felt that day-long and three-day tasks were 
inappropriate for students. They were concerned 
that this type of assessment would be inefficient, 
expensive, and very time-consuming. The group 
also voiced the same concerns as the morning group 
regarding random selection of students. 

Group members were also concerned with scoring. 
They felt the scoring on such tasks would not be 
objective or fair. They were also concerned with 
the plan to use baseline data and compare subsequent 
scores with the baseline. Several group members 
felt that this format would encourage districts to 
"cheat/ and students to perform poorly, on the 
initial administration of the tasks. 

The group also felt that the Kentucky plan for 
performance-based assessment would be very ex- 
pensive. The development and scoring of tasks and 
the hiring of assessors and scorers will cost the 
taxpayers of Kentucky a great deal of money. 

Finally, the group felt that teachers will require 
retraining in both teaching and assessment techni- 
ques. Teachers will need new and different types 
of teaching materials, hi addition, administrators, 
parents, and students will need training and aware- 
ness workshops. 

In summary, the school administrators and 
teachers attending this session were scared. They 
are afr\id that performance-based assessment will 
force them to do things for which they have not been 
prepared. They are afraid that they will be held 
accountable for results and performance for which 
they have no control. They are afraid thai they will 
not have not have the resources and training to do 
what is necessary to help students perform well on 
performance-based assessment. On the other hand, 
many educators were excited about new techniques 
and assessment which will make learning and teach- 
ing fun and enjoyable. If they just were not being 
held accountable 
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The participants in the section on foreign lan- 
guages agreed: 

• that all four sk'lls (reading, writing, 
speaking, and listening) as well as cul- 
ture need to be assessed, 

• that the tasks need to be realistic and 
very well defined, 

• that students need to be assessed as to 
how well they can function in real-life 
contexts, 

• that the assessment procedure needs to 
look at how well grammar principles are 
applied in these real-life situations and 
that students need to demon^'xate more 
than merely a knowledge of grammar 
rules, conjugations, declensions, etc., 

• that the assessment is keyed into levels 
of proficiency as defined by the 
American Coimcil on the Teaching of 
Languages, 

• that the scoring system needs to take 
into account both the complexity of the 
task and \ise, at least in part, holistic 
scoring criteria, 

• that the assessment procedure should in- 
tegrate lower levels of knowledge into 
higher levels, recognizing that language 
proficiency is to be rated on a con- 
tinuuum. 

Given illustrations of performance assessment 
in this area, what are the criteria for deciding 
whether they are good or bad? 

While keeping in mind the definition of perfor- 
mance assessment outlined above, the group focused 
on the following issues: 



If the performance assessment is to be based on 
real-life situations, then there must be oral/aural and 
reading/ writing components which are tied in with 
ongoing classroom concerns and experiences. 
Such communicative experiences will have a 
cumulative effect on student proficiency. 

There is the recognition that a sampling of perfor- 
mance will give an efficient and valid measure of 
student proficiency. 

A performance-based assessment should have the 
effect of providing valuable feedback to the class- 
room teacher and the learners as to the efficacy of 
curriculum and classroom practice in light of real- 
world expectations of language fluency. If these 
real-world expectations form the basis of assess- 
ment, then these assessment procedures will also 
satisfy the need for accountability evidenced on both 
the district and state-wide levels. 

A good performance-based test will motivate stu- 
dents to do well in that it will present students with 
interesting and meaningful tasks to be ac- 
complished. 

Some characteristics of valid and efficient tests are 
that the tests will: 

• be contextualized 

• be holistic 

• be criterion-referenced 

• be able to be successfully completed in 
the time available 

• be comprehensive and flexible enough to 
allow for the diversity of vocabulary stu- 
dents will have learned 

• \ise authentic language 

• give students the sense that the tasks 
have benefitted them 

• give students the sense that the tasks real- 
ly test what they know, i.e., that the test 
is an appropriate, adequate measure of 
learning 

• give students the chance to demonstrate 
what they can do 

The group recognized that, although the speaking 
component of any performance-based test may be 
more difficult and labor intensive to test than the 
other skills, there are methods available to test 
speaking in an efficient manner. 
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Given examples of current practice in this area^ 
in what sense have they worked well? 

The Guidelines of the American Council on the 
Teaching of Languages were cited as a proficiency 
scale underlying much current performance-based 
testing. ( ACTFJ. Proficiency G uildelines. Hast- 
ings -or -Hudson, NY: ACTFL Materials Center, 
1986.) 

Are there implications of performance assess- 
ment in terms of fairly assessing the different 
genders^ racial and ethnic groups^ and social 
classes? 

The group discussed the issue of cultural diversity 
and performance outcomes in terms of: 

• lexica, concrete topics to be tested, etc. 

• the fact that a foreign language can 
serve to motivate students due to the feet 
that students tend to begin with zero 
knowledge of the language. This fact 
can place students from diverse back- 
grounds on a more equal level. 

• by teaching the culture of the foreign lan- 
guage, students should become more sen- 
sitized to cultural diversity. 

What are the structural barriers to using these 
activities in the schools? Are there areas in which 
training will be required? 

The group felt that there were no structural barriers 
per se to implementing these new kinds of testing 
procedures, but felt that: 

• teachers would need to be retrained in 
terms of the communicative 
methodologies which are performance- 
oriented. 

• teachers will need to have access to the 
materials necessary to focus students on 
real-life situations. 

• teac^-^rs will also need to be trained in 
methods that will allow learners to use 
the language in real-life groupings, i.e. 
with one or more conversational 
partners and that will optimally involve 



each student in using the target lan- 
guage. 

• teachers may also need to have extra 
training to increase their own foreign- 
language skills, especially their oral 
skills. 

• the Commonwealth of Kentucky needs 
in-service standards reflecting the skill 
levels necessary for teachers in a 
proficiency-based, communicative class- 
room. 

• the Commonwealth of Kentucky must 
provide the financing necessary for 
workshops and other training needed to 
bring teachers up to the necessary stand- 
ards. 
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We now turn our attention to the initial 
tasks involved in planning for the im- 
plementation of mastery learning. The 
first of these tasks is to set out, in specific terms, 
what students are expected to leam. These expec- 
tations are generally referred to as learning objec- 
tives y and they are the main topic of this chapter. 

Learning objectives describe the skills and abilities 
students are to acquire as a result of our teaching. 
The process of specifying them requires that a series 
of important decisions be make about what is essen- 
tial for students to leam and at what level that 
learning should occur. While these decisions are 
fimdamental to the use of mastery learning, they also 
add precision to procedures for evaluating students* 
learning. In mastery learning they further serve as 
a basis for developing formative tests and planning 
feedback and corrective activities. 

To assist teachers in making these decisions, many 
commercial publishers list the particular learning 
objectives that their materials have been designed to 
help students attain. Although these vary widely in 
detail, most teachers find them helpiiil in planning 
instruction. Some commercial materials also in- 
clude tests for checking on students* leamiiig 
progress. These, too, can be useful and can help 
reduce the amount of preparation time required of 
teachers. Unfortunately, though, few commercial 
materials are universally applicable. Refinements 
and adaptations are usually necessary to meet the 
needs of particular groups of students or the instruc- 
tional preferences of different teachers. Therefore, 
along with considering the decisions that need to be 
make in outlining learning objectives, we will in this 
chapter also consider procedures for reviewing and 
adapting objectives and materials that may be al- 
ready available. 

THE IMPORTANCE OF A STRUCTURE 
FOR LEARNING 

Generally when scholars or researchers reach a 
certain level of sophistication in a subject, they are 
able to see definite relations among the ideas and 
concepts in that subject. These relations help them 



understand more complex phenomena and aid them 
in conducting further study. In many cases, cur- 
riculum writers, most of whom are also experts in 
the subject, make the assumption that the structure 
of the knowledge in that subject is synonymous with 
an appropriate structure for teaching and learning 
that subject. Unfortunately this is not always true. 

For example, several modem mathematics and 
science curricula were based upon the belief that if 
the sophisticated organizing principles that experts 
foimd so helpful could be provided to young people 
learning the subject, the young people would find 
the subject much easier to leam. It was quickly 
discovered, however, that while these organizing 
principles are very useful for specialized scholar- 
ship, they are not always useful in helping the 
majority of students leam the subject. As B.S. 
Bloom, J.T. Hastings, and G.F. Madaus point out: 

The usefulness of a stmcture for learning 
has to do with the ability of students to 
comprehend it and use it as an organizing 
factor in their learning. There is no relation- 
between the usefulness of a structure for 
scholars and its usefulness (and meaningful- 
ness) for students. (1971. p. 12). 

It is true that students leam more easily when 
provided with a structure that helps them relate 
various aspects of the subject. Such a stmcture also 
helps students gain deeper meaning from what migt £ 
otherwise be a ^'\rge number of unconnected 
specifics. Research studies, particularly ihi.s^ on 
the use Oi^ "advanced organizers" in instruction 
(Ausubel, 1963, 1978), have shown that ideas are 
more easily grasped and remembered when leamed 
in relation to one another ralher than in isolation.* 
A stmcture for learning should thus provide students 
with a mechanism they can use to better understand 
the instmction and to organize the concepts they are 
leaming. It is important to keep in mind, however, 
that a stmcture for leaming is based primarily on 
pedagogical considerations and may not be the same 
as an expert's or scholar's view of the field. 

ORGANIZING LEARNING UNITS 

Developing an appropriate stmcture for leaming 
generally involves three elements. 
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1. The final learning goal to be attained must be 
specified. This goal is usually a competent learner 
who has truly mastered those things we set out to 
teach. 

2. The final learning goal must be analyzed to 
identify the steps that are necessary to reach the 
goal. 

3. The steps must be ordered in an appropriate 
sequence to facilitate learning and provide for 
steady and regular progress toward the goal. 

Although these three elements may seem impUcit 
in all teaching, one or more is often neglected. For 
instance, the daily burdens of teaching can some- 
times distract a teacher's perspective from the final 
learning goal and, as a result, teaching efforts lose 
their focus and cohesion. Similarly, concentrating 
solely on the goal without careful attention to the 
separate steps required to reach that goal can result 
infrustration for teachers and students alike. Both 
the goal and the sequence of steps required to reach 
that goal need to be kept in mind if teaching and 
learning are to be effective. 

The process of analyzing a learning goal and then 
organizing the steps necessary to reach that goal is 
a natural part of most teaching and learning ac- 
tivities. For example, consider how you might go 
about teaching a child to play tennis. You would 
probably begin with a mental picture of the child 
approaching the ball, swinging smoothly, and 
returning the ball to the other court. This is the 
final goal you would hope to attain at the end of the 
learning process. From that menul picture you 
would begin by dividing the components of that final 
learning goal into various steps. You would probab- 
ly think about adjusting the racket to the child's size 
and strength; adjusting the child's grip for backhand 
and forehand returns; telling the child about the 
importance of watching the ball; and showing the 
child how to move to the ball in order to make the 
backswing, return and follow-through, and then 
recover for the next return. You would also need 
to demonstrate serving. In addition, you would 
want to explain the rules of the game and how to 
keep score. Given this breakdown you would then 
decide upon an appropriate sequence of learning 
steps. You might decide to order the steps in terms 
of difficulty or complexity. The most basic ele- 
ments, such as watching the ball, would be presented 
before more complicated steps, such as an ap- 



propriate follow-through. Then, as you begin 
teaching, you would try to be aware of any special 
problems the child may be having and would try to 
correct them as they appear. In addition, you would 
probably make a point of rewarding the child when- 
ever possible and providing reassurances at other 
times. 

This example illustrates the process that takes 
place in most effective teaching and learning situa- 
tions. The learning goal, or what is sometimes 
referred to as the summative goal, is first analyzed 
in terms ofthe parts that need to be mastered. Those 
parts are then organized and arranged in an ap- 
propriate sequence of learning steps. Care is taken 
to ensure that each of these steps is mastered while 
progressing toward that final goal. 

Similarly, to begin the use of mastery learning, it 
is first necessary to identify the final learning goal 
and to define the specific steps that need to be 
mastered in order to reach that goal. Many teachers 
do this regularly as a part of their instructional 
planning. That is, they start out with a mental 
picture of a competent learner at the end ofthe 
course of instruction and then divide the material 
that must be learned over the year or term to reach 
that goal into smaller components or steps. Each of 
these steps is considered a learning unit. 

The delineation of learning units is somewhat 
arbitrary in many cases. IdeaUy, learning units 
should be determined by natural breaks in the sub- 
ject material or by content elements that make a 
meaningful whole. Thus each unit might not cover 
exactly the same amount of content. Textbook 
publishers usually divide the content of a particular 
subject in accordance with these natural breaks. For 
example, chapters in textbooks often represent ap- 
propriate learning units. 

Another critical element to keep in mind in deter- 
mining learning units is instructional time. A learn- 
ing unit should contain material that can be 
presented in about a week or two of classroom time. 
Generally, learning units at the high school or 
college level are longer and cover more material 
than learning units in the elementary grades. A unit 
in a high school course may last two weeks or 
slighdy longer and cover twenty or thirty important 
objectives. An early elementary school unit, on the 
other hand, seldom lasts more than a week and may 
cover a single skill or fragmented and generaliza- 
tions will be difficult to buJd. If too long, how ver, 
students who fall behind because of particular l(*am- 
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ing problems may have great difficulty catching up. 
Thus not only the content but also the pace of 
instruction and the kinds of students involved in the 
learning need to be considered in determining ap- 
propriate learning units. 

TABLES OF SPECIFICATIONS 

Once the sequence of learning units is clearly 
delineated, the next task is to specify the learning 
objectives of each unit. To do this, it is necessary 
to first identify the new material that will be 
presented in each unit. Many teachers develop 
detailed outlines of the new material they plan to 
present and often consider this material to be the 
objective of the unit. But while deUiled outlines of 
new material can be very useful in teaching, they 
say nothing about what students might be expected 
to do with that material. Therefore, in specifying 
learning objectives for a unit, it is important to 
consider not only the material or content students 
are expected to learn, but also the specific behaviors 
they are to attain in relation to that content. These 
behaviors indicate the ways we would like students 
to be able to think, act, or feel about the material, 
about themselves, about others, and so forth. That 
is, what students are expected to do with the material 
is just as important a consideration as the material 
itself. 

A useful and efficient way to outline the new 
material and the behaviors students are expected to 
attain is to construct a simple two-dimensional table. 
This table is sometimes called a "mastery chart" or 
a "matrix of content and behaviors. " More general- 
ly, it referred to as a table of specifications. 

A table of specifications is basically an outline of 
the learning objectives for a unit. As such it serves 
two important functions. First, it helps add 
precision and clarity to teaching. The information 
on the table should be precise enough to convey 
exactly what is intended in the instruction, and clear 
enough that students can be helped to fully under- 
stand what they are expected to learn. Many 
teachers go through a similar specification process 
as a regular part of their class preparations. But 
often what is e .^.ected of students is never make 
clear until the time they are evaluated. In these 
instances, students are forced to guess what is 
important and what is inconsequential. While some 
students are very good at guessing v/hat they are 
expected to learn, many others are not. 



For those who are not, learning soon becomes a 
very frustrating experience. Specifying the objec- 
tives for learning precisely and clearly, and com- 
municating these objectives to students, not only 
eliminates much of this guesswork but also helps 
organize and focus teaching and learning activities. 

The second function of a table of specifications is 
to serve as a guide for consistency between learning 
objectives and procedures for checking on students* 
learning progress. Although this kind of consisten- 
cy is very important for learning, it is often 
neglected or given only cursory attention. For 
example, many classroom teachers stress that they 
want their students to develop higher cognitive skills 
such as the ability to make applications, analyses, 
or syntheses. However, the vast majority of class- 
room tests tap onl> those skills that are easiest to 
assess, such as knowledge of the definitions of terms 
or specific facts. A table of specifications can be 
used as a guide in preparing a wide variety of tests 
and evaluation procedures. In this way, it can help 
to guarantee consistency between important objec- 
tives and procedures for checking on students' learn- 
ing (see, for example, the sample tables in chapters 
9 through 14). 

STEPS IN DEVELOPING A TABLE OF 
SPECIFICATIONS 

The first step in preparing a table of specifications 
is to determine what new material or content is 
introduced in the learning unit. That is, what are 
the new terms, facts, relations, procedures, and so 
forth that are explained, defined, illustrated, or 
otherwise presented in the unit? Usually textbooks 
and other learning resources are relatively clear in 
signaling when new material is being introduced. 
Changes in print or color, comments in the margins 
(particularly in teachers* guides), and summaries at 
the end of chapters often identify new content. 

The next step is to determine the particular student 
behaviors that should be paired with the new 
material or content. That is, what are students 
expected to do with the new content? Will students 
be required to simply remember or recall the new 
content, or will we want them to be able to use it ta 
a new or different way? In specifying these 
decisions, many teachers find it useful to classify 
the new elements of content according to some of 
the categories in Taxonomy of Educational Objec- 
tives, Handbook I: Cognitive Domain (Bloom, 
Englehart, Furst, HiU, iSc Krathwohl, 1956). These 
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categories represent a hierarchy of levels that differ 
in terms if difficulty and complexity. The lowest 
levels represent the simplest kinds of learning, while 
higher levels represent more advanced cognitive 
skills. The categories that are most useful in a wide 
variety of subject areas are shown in the table of 
specifications in Figure 2.L These levels include: 

1 .Knowledge of terms. Terms are the new words 
or phrases that students are expected to learn. They 
may be expected to define these terms, recognize 
illustrations of them, determine when they are used 
correctly, and/or recognize synonyms. Examples 
of such terms are product in a mathematics unit on 
multiplication or photosynthesis in a science unit. 
The "knowledge" level here is simple recognition 
or recall, and is generally considered to be the 
simplest level of student learning. All new words 
or phrases to be introduced and explained in the unit 
are listed in the table under "Terms." 

l.Knowledge of facts. Facts are the specific types 
of information that students are expected to remem- 
ber. In general, facts are particular details that are 
important in their own right or are essential for other 
kinds of learning. Facts include names of persons, 
events, operations, or other kinds of specific infor- 
mation. Students may be expected to recall par- 
ticular facts and/or remember the correct fact when 
asked about it in a relatively direct manner. Ex- 
amples of facts are "The Declaration of Inde- 
pendence was signed on July 4, 1776" and "Stephen 
Crane is the author of The Red Badge of Courage. " 
Any new facts in the unit should be listed in the 
column labeled "Facts." 

3. Knowledge of rules and principles. Rules and 
principles concern specific patterns or schemes that 
are used to organize the major ideas of a subject. 



Generally they bring together a number of facts or 
describe the interrelationships among a number of 
specifics. Students may be expected to know a rule 
or principle, to remember an illustration of it used 
during instruction, and/or to recall situations in 
which it was applied. In most cases, rules and 
principles are more difficult to learn than are terms 
or facts. However,, it is important keep in mind that 
this category deals only with the knowledge of rules 
and principles, not with their application. The 
"commutative principle" in mathematics and the 
"rules for subject/verb agreement" in grammar are 
examples of rules and principles. New rules and 
principles in a unit should be listed in the table. 

^.Knowledge of processes and procedures. In 
many subjects, students are expected to know the 
particular steps involved in a certain process or 
procedure. Frequently it is important that these 
steps be recalled in a specific sequence. For ex- 
ample, students may be expected to know the ap- 
propriate order of steps in a mathematics 
problems-solving task and/or the sequence of events 
necessary to enact legislation in Congress. Proces- 
ses and procedures may involve a number of terms 
or facts and are typically difficult for students to 
learn. Any new processes or procedures should be 
included in the table. It is important to keep in mind, 
however, that these should be the processes and 
procedures we want students to learn, not the in- 
structional processes or procedures we plan to use 
in teaching the unit. 

S. Ability to make translations. Translation invol- 
ves the transformation of a term, fact, rule, or 
process from one form to another. In making a 
translation, students express particular ideas in a 
new way or take phenomena or events in one form 
and represent them in an equivalent form. Students 
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may be expected to recognize new illustrations of a 
term, feet, rule, or other matter and/or to determine 
whether a new illustrations is appropriate. In 
general, then, students employ translation when 
they put an idea in their own words or recognize 
new examples of what they have already learned. 
Translation in this sense should not be confused with 
translation in foreign language instruction, though 
they are similar. (The special considerations 
regarding foreign language instruction are discussed 
in detail in Chapter 14.) The specific translation 
skills students will be expected to learn should also 
be listed in the table. 

6, Ability to make applications. Application is the 
use of terms, facts, principles, or procedures to 
solve problems in new or unfamiliar situations. 
Students are expected to use ideas or concepts 
learned in one context to solve a problem presented 
in a new context. It is important to remember, 
however, that if the problem is one students have 
encountered previously except that new data are 
substituted, then the behavior called for is really a 
translation rather than an application. Because of 
this , it is difficult to name specific examples. An 
application for one teacher might be a translation for 
another, depending upon differences in the ex- 
amples used in class presentations or practices. In 
making an application, students must first recognize 
the essentials for the new problem; determine the 
facts, rules, procedures, and so forth that are 
relevant; and then use these to solve the problem. 
The ability to make applications is a feirly complex 
behavior and is the highest level of learning objec- 
tive used by most teachers. The application skills 
students are expected to develop m a unit should be 
included in the table. 

1 , Skill in making analyses and syntheses. An even 
more advanced behavior is that of making analyses 
and syntheses. Analyses generally involve the 
breakdown of concepts into their constituent parts 
and the detection of the relationships among those 
parts. Syntheses, on the other hand, involve the 
putting together of elements or concepts in such a 
way as to develop a meaningful pattern or structure. 
Syntheses often call for students to develop creative 
solutions within the limits of a particular problem 
or methodological framework. Because of the com- 
plexity of these behaviors, analyses and syntheses 
are typically considered only in advanced or higher 
level classes. However, some teachers feel that 
these skills are very important and try to include 



practice in these kinds of activities for students at 
all levels. Examples of analyses are "distinguishing 
facts from opinions in a communication" and "iden- 
tifying conclusions and supporting statements. " An 
example of synthesis is "Write a paragraph that 
organizes a set of ideas and statements." Any 
analyses and syntheses included as a part of students' 
learning should be described in the table. 

In general, teachers have little difficulty outlining 
learning objectives in terms of these categories of 
behavior. Most find that new material in a imit can 
usually be classified in one or another of the first 
four knowledge level categories. Few textbooks or 
other learning resources present material at more 
advanced levels. Decisions regarding more ad- 
vanced behaviors must therefore be made apart from 
the materials. 

At times , some confusion does arise as to whether 
a particular objective should be listed under one or 
another of two adjacent categories. For example, 
an objective might be considered a fact by one 
teacher and a principle by another; or a translation 
to one teacher might be considered an application 
by another. But differences such as these are usually 
of Uttle consequence. So long as the objective is 
included in the table, whether it is in one column or 
the next adjacent coliunn makes little difference. 
Major distinctions, for instance, between the 
knowledge level and the application level, are im- 
portant, however, and need to be carefully con- 
sidered. 

RELATIONSHIPS AMONG LEARNING 
OBJECTIVES 

Besides listing the new elements of content and 
what students are expected to do with that content, 
the table of specifications can also be used to il- 
lustrate relationships among content elements. For 
instance, knowing the definition of a term may be 
necessary in order to understand a fact pertaining to 
that term. Or, knowing two or three fects may be 
essential in understanding a particular procedure. 
These relationships can be illustrated on the table by 
drawing connecting lines between these elements. 
For example, lines might be drawn from two or 
three new terms to a fact that incorporates these 
terms. A line could also be drawn from knowledge 
of a particular procedure to the application of that 
procedure. Many teachers find that drawing these 
lines helps them keep these relationships in mind so 
that they can be developed for students. An example 
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illustrating these relationships for an elementary 
social studies unit is shown in Figure 2.2. 

For the vast majority of teachers, developing a 
table of specifications is not only useful, it is also 
very revealing. The table enables them to view the 
new elements and their relationships in a very 
compact form. By making tables of their own or by 
carefully reviewing prepared tables, teachers can 
add greater precision to their teaching and can more 
closely match presentations of unit material with 
learning objectives. A table of specifications can 
also help reveal possible gaps in the instruction. It 



particular subject. Some subjects, such as social 
studies, may have no or very few rules and prin- 
ciples. Similarly, some units within a subject, such 
as mathematics, may not include any new terms. 
Thus it is not unusual to find that some of the 
categories in a table of specifications for a particular 
subject or unit are omitted or left blank. Only those 
categories that are usefiji in listing the new material 
and specifying the behaviors expected of students 
need to be included. These subject differences can 
be seen by inspecting the tables of specifications in 
Chapters 9 through 14. 
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Figure 2.2 Portion of table of specifications showing related dements of content in an eietnentary social studies unit. 



can show where important elements may have been 
neglected or where relationships between elements 
need to be pointed out. Furthermore, a table is 
useful in constructing instruments to check on 
students* learning progress since it provides a guide 
to what elements of content should be tested and 
what student behaviors should be assessed. These 
considerations are discussed more extensively in the 
next chapter. 

SUBJECT AREA DIFFERENCES 

Learning objectives from nearly any subject can 
be outlined according to the categories shown on the 
table of specifications in Figure 2. 1. However, all 
of these categories may not be applicable to a 



The difficulty of preparing a table of specifications 
also varies depending upon the subject. Some sub- 
jects, such as mathematics and science, lend them- 
selves quite easily to this format. It is usxially not 
difficult to identify the new terms, facts, rules, 
and/or procedures that are part of a mathematics or 
science unit. However, subjects such as reading, 
composition, and language arts are not as easily 
categorized. Special adaptations are sometimes 
necessary for these subjects, as can be seen in the 
language arts unit in Chapter 11. StUl, it remains 
important to precisely specify the new ideas or 
concepts to be presented, and what students will be 
expected to do with that iuformation. For instance, 
in teaching students how to make inferences, it is 
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important to specify the various components that go 
into making inferences and how to distinguish be- 
tween a student who can make appropriate inferen- 
ces and one who cannot And more importantly, 
we must consider how to help students who cannot 
make inferences to acquire that skill. Issues such as 
these need to be given serious consideration when 
outlining learning objectives for a unit in reading or 
composition. This kind of specification greatly 
enhances the organization of instruction and helps 
students be more focused in their learning efforts. 

TABLES OF SPECIFICATIONS AND THE 
TEXTBOOK 

For most teachers, the basic textbook used in 
teaching serves as a primary guide to the new 
material introduced in a learning unit. Some 
teachers do supplement the material presented in the 
textbook with information from curriculum guides, 
other textbooks, learning kits, or special classroom 
activities. But, generally, the course textbook is the 
principal source of new material presented in each 
unit. 

The vast majority of textbooks present new 
material at strictly the knowledge level. In fact, few 
textbooks compel students to do anything other than 
know or re^^U the information presented. Oc- 
casionally at the end of a chapter there will be a 
section entitled "Questions for Discussion" or "Spe- 
cial Activities" that requires students to use the 
material presented in the chapter in a new or dif- 
ferent way. But since so few textbooks go beyond 
the knowledge level when presenting new informa- 
tion, often only the knowiedge-level categories are 
filled when a textbook is used as the principal guide 
in initially preparing a table of specifications. 

Nevertheless, most teachers are interested in a 
wider range of student behaviors than simply those 
associated with basic knowledge. Many want their 
students to develop higher cognitive behaviors as 
well, to use what they are learning in new and 
creative ways. In these instances, it is necessary to 
go beyond the textbook in outlining learning objec- 
tives. That is, although the textbook can serve as 
an excellent guide to the content of a learning imit, 
it usually does not clearly specify what students 
might be expected to do with that content. These 
decisions must be made by curriculum planners or 
by individual teachers. In preparing tables of 
specifications or reviewing tables that have been 
previously developed teachers need to consider the 



types of behaviors they want their students to attain 
as a result of their teaching. 

TABLES OF SPECIFICATIONS AND 
LESSON PLANS 

The process of preparing a table of specifications 
can greatly add to the clarity of lessons and class 
presentations. In fact, many teachers use the table 
as a guide during teaching acti^•ities. But for many 
others, a table of specifications by itself is insuffi- 
cient as a lesson plan. 

Particularly at the elementary level, most teachers 
prefer to teach from lesson plans that are fairly 
detailed. These plans describe not only what is 
going to be taught but also how it will be taught. In 
preparing these plans, teachers often list the pivotal 
questions they plan to employ in their presentations 
or die guiding questions they want to use to stimulate 
discussion. Sample problems may be illustrated and 
specific examples may be hsted. While details such 
as these are extremely useftil in presenting a lesson, 
they are usually not included in a table c^^ specifica- 
tions. A table of specifications is usually quite brief 
and will describe the learning objectives for an entire 
unit in a page or two. In essence, it is designed to 
address two questions: (1) What do I want my 
students to learn? and (2) What do I expect my 
students to be able to do with what they have 
learned? The first of these questions concerns the 
content; the second concerns student behaviors in 
relation to that content. Note, however, that both 
of these are "what" questions. The "how" questions 
are usually not addressed in the table. 

It is true that the types of learning outcomes 
described by the objectives outlined in a table of 
specifications suggest different approaches to teach- 
ing. For example, while drill and recitation may be 
appropriate techniques for teaching knowledge of 
terms and facts, they are inappropriate for teaching 
students the skills involved in making analyses and 
syntheses (Gagne, 1977). The inclusion of both of 
these levels of student behavior on a table of 
specifications thus imphes that different approaches 
to instruction will be used. But, generally, teachers 
who want a more detailed plan that iliastrates 
specifications with lesson plans that include ftulher 
directions and information. Such lesson plans typi- 
cally describe the activities to be ^ised each day or 
in each class session, and they directly address the 
question of "how" these objectives are to be ac- 
complished. 
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BENEFITS OF A TABLE OF 
SPECIFICATIONS 

Outlining learning objectives and preparing tables 
of specifications clearly illustrate the importance of 
the decisions make by teachers in using mastery 
learning. Even tiie process of reviewing tables 
prepared by other teachers or curriculum specialists 
requires that carenil thought and consideration be 
given to what students will be expected to learn. 
Although the importance of these decisions are a 
fundamental part of all teaching. Judgements about 
what should be taught, what students should learn, 
and what students should be expected to do with 
what they have learned are made by teachers on a 
daily basis. Developing a table of specifications 
simply compels teachers to be very conscious of 
those judgements and decisions, and to make them 
in a very explicit way. As a result teaching activities 
can be better organized and much more focused. 

The process of preparing a table of specifications 
may seem somewhat difficult and cumbersome at 
first. Generally this is because making these judge- 
ments and decisions in. such a specific manner is an 
unfamiliar experience. The process requires a 
perspective toward learning objectives and instruc- 
tional goals that is new for many teachers. However, 
within a very short time most find that the process 
becomes much easier. The time required to prepare 
a table of specifications for a second unit is usually 
about half that required for the first. A table for a 
third imit requires even less time. 

In addition^ most teachers discover that teaching 
becomes somewhat easier as a result of this process. 
In the long run, clearly specifying what students are 
to leara and what they are expected to do can save 
a teacher valuable time and also provide a sense of 
accountability. Furthermore, being well organized 
allows a teacher to concentrate more fully on ways 
to best present new material and on different 
methods for involving students in the instruction. In 
this way, a table of specification's is similar to a map 
used by travelers. While it does not limit the 
pathways that can be taken, it does enhance the 
efficiency of the traveling, the enjoyment of the 
travelers, and the likelihood that all will successfully 
reach a particular destination. 

SUMMARY 

Having a clear notion of what is to be taught and 
what sDidents are expected to learn is essential in 
implementing mastery learning. This is usually 



accomplished by outlining specific learning objec- 
tives. The tasks involved in this process are: 

1. Identify specific learning units. Learning units 
are typically determined by the natural breaks in the 
subject content and usually represent the material 
covered in about a week or two of instructional time. 

2. Determine the content to be presented in each 
unit. List the new terms to be introduced, the facts 
to be presented, the rules and principles to be 
explained, and the processes and procedures to be 
discussed. 

3 . Determine what students are to do with the new 
content in each learning unit. Indicate whether 
students will be expected to know or recall the new 
material, translate it from one form to another, apply 
it in a new or different situation, or analyze it in a 
new way. 

4. Prepare a table of specifications summarizing 
these decisions, A table of specifications outlines 
the content of a unit and the learning outcomes 
expected of students in relation to that content. It 
can also serve as a guide in planning instruciion and 
evaluating students* learning, 

ACTI\TriES 

1, In a familiar subject area, preferably one you 
teach, identify a learning unit. If you do not present- 
ly teach, crvnsider the material presented in this 
chapter as a learning unit. 

2, Construct, using the general format shown in 
Figure 2.1, a table of specifications for the unit, 

3, Have a friend or teaching colleague review, 
both the imit and your table of specifications, asking 
of the table, Are these the same elements of content 
I would identify a important in this unit"? and "Are 
these the same behaviors I would require of stu- 
dents?" 

4, Discuss any differences that arise through this 
review. Although such differences are likely to be 
few, their clarification can add greatly to the 
precision and completeness of a table of specifica- 
tions. 
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Fifteen attended the morning session including 
the two leaders. There seemed to be some 
distribution geographically and professional- 
ly with a mix of administrators and teachers from 
various school districts in the eastern half of the 
state. 

Linda Levstik opened the session with a round of 
introductions and a reaction to the remarks made by 
Dr. Gary Phillips about the NAEP program. Linda 
explained that the NAEP Social Studies test is not 
performance based. She then showed samples of 
the performance assL ment items which are under 
consideration by the state Social Studies Task Force 
for grades 4 and 8. Discussion center on reactions 
to the items shown and the following questions: 

What is performance assessment? 

This question raised concern among participants 
about potential problems with performance based 
assessment. One participant mentioned that while 
it is laudable that performance based tests aim to be 
authentic, it is important to remember that many 
activities in the classroom cannot be assessed. 
Others expressed concern about how to design tests 
that capture students' knowledge of and ability to 
interpret content and simultaneously test process 
skills. They also wondered how schools might be 
compared under this system. 

Other potential problems mentioned include the 
impact of mainstreaming on curricular changes and 
the difficulty involved in creating objective, valid, 
reliable ^xsts. One participant expressed concern 
about administration of the test. If these tests are 
given over several days, validity could be jeopard- 
ized since students would have opportunities to seek 
assistance. Another participant felt it would be 
difficult to test both content and process skills with 
this measurement system. 

One participant suggested that policy-making 
bodies should focus on examining the knowledge 
base which undergirds the Social Studies cur- 



riculum. This person also suggested that the linkage 
betweer pedagogy and content should be considered 
as well as how to integrate Social Studies with other 
areas of curriculum 

What are the criteria for assessing 
assessment? 

Participants agreed that the most important 
criterion is to devise a way to test process skills and 
content together. They also diought a careful deter- 
mination of which content areas should be tested is 
a critical consideration since Social Studies encom- 
passes such a broad area of topics and issues. 
Several felt that specific goals should be determined 
and a variety of tests given in order to assess all 
aspects and levels of learning from retention to 
interpreUtion of subject matter. 

How does performance assessment 
compare with traditional "paper and 
pencil" tests in terms of validity or 
efficiency? 

Participants agreed that multiple choice tests are 
more efficient. The utility of performance assess- 
ment tests is yet to be determined. Participants felt 
they would need to have more information in order 
to respond adequately to this item, and concern was 
expressed about the lack of information which has 
been provided, to date, about implementation. 

Given examples of current practice in 
this area, in what sense have they 
worked well? 

Participants felt that, at this point, they do not have 
enough experience to determine how performance 
assessment will work; however, the following com- 
ments were made about the issue in general. 

One participant pointed out that many reform 
Lnitiati>'es can be traced to John Dewey's woric and 
the Progressive movement in education in the early 
I900*s. Another participant reiterated the impor- 
tance of establishing meaningful goals and adapting 
the curriculum to fit these goals. Someone else 
suggested that in doing so careful consideration 
should be given to the selection of material used for 
instruction, especially textbooks. Another par- 
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ticipant suggested that those designing assessment 
measures need to work closely with the State 
Textbook Commission, because the Commission, 
through the selection of classroom material, sets the 
curriculum. She also said that it will be important 
to acquaint the public with changes in assessment 
and the resulting change in pedagogy. 

What are the imp'kxations for fairly 
assessing different gender, racial, 
ethnic groups and/or social classes? 

Discussion of this issue centered around Dr. 
Phillip's comment that minority students do not 
score well on performance assessment type items. 

The point was made that students from varied 
ethnic and social groups tend to score poorly on the 
Social Studies performance assessment items, bu. 
black and hispanic students score very poorly. Par- 
ticipants speculated that the reason for this may be 
lack of experience with reading and writing. One 
participant suggested that tests should account for 
differences in learning styles. 

Seven participants attended the afternoon ses- 
sion including the two leaders. As with the 
morning session, this began with a round of 
introductions and orientation by sharing two 
samples from the state task force's work. The 
discussion centered on reacting to these items and 
responding to the following questions. 

Do the activities defined, illustrated, 
and discussed in the morning session 
meet the requirements of the law? 

Generally, participants felt that teachers are 
engaged in the type of activities that fit with perfor- 
mance assessment although sokGq noted that not all 
teachers create activities that /ould fit. Participants 
suggested that, in order to create a fit, extensive 
training would be needed. 

What are the structural barriers, if any, 
to using these activities in the schools? 

One barrier is that Social Studies is often left out 
of the curriculum, and the Social Studies curriculum 
for the primary grades tends to be weak. A second 
barrier, according to participants, is parent involve- 
ment. Parents want justification for changes in 
testing and curricula, and if they are not satisfied 



with the reasons for change, they may voice objec- 
tions. Departmenfalization was the thl a barrier 
mentioned. Schools need to integrate social studies 
into other aspects of the curriculum and vice versa. 
Textbooks are also problematic because teachers 
tend to use them as a crutch. Teachers need to 
consider using a wide range of material and ac- 
tivities to interest students in learning about Social 
Stud^^/S. A fourth, and final, barrier is the amount 
of content to be covered in the Social Studies 
curriculum. Participants felt that content would 
have to be carefully selected. 

What are the costs of implementing 
these new activities in terms of both 
material and labor? 

Implementation will be costly in terms of both time 
and labor, according to the participants. Social 
Studies requires use of a wide range of material, and 
the amouTit of content requires a great deal of time 
to cover. In addition, longitudinal studies will be 
needed to determine the impact of performance 
assessment as students progress through the grades. 
One participant pointed out that it would take half 
of an average teaching career to study one group of 
students grades K-12. 

Are there areas in which training will be 
required? 

Extensive training is needed for implementation of 
the reform initiative in the urea of St^iii Studies. 
Participants felt that teachers need to hav^: time away 
from their school duties to use performance type 
tasks and discuss practices which would help stu- 
dents learn. One participant suggested that a very 
effective form of training Wv . : i be to have consult- 
ants teach a class, demonstrating methods over an 
extended period of time. 
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From the discussions in the various sessions 
and presentations given at the conference, it 
appears that professionals in the schools still 
do not know very much about how performance 
assessment will be done in Kentucky. Yet we 
only have a short time to prepare for performance 
assessment in the Kentucky schools. The lack of 
direct and simple implementation suggest that part 
of the implementation include funding for training 
for teachers and school administrators. 

Based on the experience in this and other states and 
in NAEP, certain useful points can be made as we 
prepare for performance assessment in our schools. 
KERA mandates changes in school assessment from 
an emphasis on relatively artificial demonstrations 
of knowledge (like paper-and-pencil multiple- 
choice tests) to the seemingly more practical 
performance based assessment. Yet assessing 
school performance using so-called realistic and 
practical tasks is not as obvious or as simple as it 
appears on first glance. 

Further, what this means for teaching and the 
classroom is different from one field to another. For 
example, physical education, vocational education- 
al, language education, and much of art education 
have been largely peformance-based for some time. 
Other fields like math, social studies, and science 
will have to make more significant changes in how 
they view assessment. So the impact should be 
different in different areas of the curriculum. 

Tasks in a performance assessment should be 
varied. Some might be very practical, like balanc- 
ing a checkbook or creating a recipe for a blueberry 
cobbler. Others might be paper-and-pencil and be 
more dependent on language and thinking skills. As 
Gary Phillips suggests, there might even be an 
appropriate place for the venerable multiple-choice 
item. 

The exact nature of the assessment task is deter- 
mined by the goals stated for the curriculum and the 



assessment. Traditional assessment methods stress 
knowledge. Performance assessment stresses 
broader and more comprehensive responding. For 
example, after a biology unit on ecosystems, a 
student might be asked to set up a self-sustaining 
ecosystem in an aquarium. This task would ob- 
viously test more of the student's knowledge than 
would a multiple-choice item asking the child to 
identify which of five choices contains things neces- 
sary to sustain life. 

The performance-based item would provide the 
student with a chance to show a more integrated 
understanding of the task. However, it is expensive 
in terms of staff and the child's time, and in other 
ways. Further, scoring of such a task is not as clear 
as scoring of traditional tests. 

Another aspect is the cost. A true performance 
assessment of every child over every facet of the 
K-12 curriculum is beyond the total budget 
capability of Kentucky. This means that some sam- 
pling plan is necessary. Sampling will probably 
occur in terms of what is assessed from the cur- 
riculum, who is assessed on terms of students, 
schools and perhaps even school districts, and 
methods of assessments. 

The law calls for test equating and establishing 
baselines of performance. Yet there are difficut 
technical problems that must be dealt with to ac- 
complish this. The seeming simplicity of perfor- 
mance assessment can encourage avoidance of the 
thorny technical aspects. However, an assessment 
system must meet rigid criteria established for 
educational assessment systems if it is to be 
depended upon in the manner suggested by the the 
Kentucky Education Reform Act. 

Schools will do continuous assessments of students 
performance and progress. These assessments may 
include a variety of methods of illustrating student 
progress. A very common one is the establishment 
of portfolios for each student. A portfolio is a 
folder that contains all the work the student has done 
(such as home work, traditional classroom tests, 
essays, and projects). Another, is to use group 
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problem solving approaches as well as traditional 
classroom tests and assignments to gauge perfor- 
mance. In fact, different methods might be ap- 
propriate depending on the goals stated for each 
sector of the ciirriculum, and goals for tying together 
various sectors, such as science and math, mathe- 
matics and language arts, etc. could be developed. 

The continuous assessment that will be done in the 
schools will be important in the overall assessment 
of the school program. Further, teachers and ad- 
ministrators should have something to say about the 
design of this aspect of assessment. 

Performance assessment will not be put into place 
all at once. The law suggests that this procedure 
will be done in stages. The first stage wiU be an 
interim testing program in 1991-92 in which a 
sample of students will be assessed to establish 
baseline data. A more complete implementation 
will follow in successive years. Regional assessment 
centers may be established to which samples of 
students might be brought for assessment activities. 
This implies that some aspect of the performance 
based assessment wiU be done outside of the regular 
school program and involve samples of students. 

Other critical issues include consideration of in- 
dividual students propensity to perform in different 
ways, regional differences in expression and ex- 
perience, culture-fairness of certain kinds of tasks, 
and fairness to special populations. In addition, the 
match between the curriculum used in the classroom 
and the performance assessment method will require 
considerable attention. The amount of time school 
teaching staff have to prepare for and present assess- 
ment tasks, and their role interpreting and scoring 
should be explored as weU. 

Six goals have been established by the reform act 
for areas of student performance that will shape the 
school curriculum and the assessment. The goals 
are: (1) The application of basic communication and 
mathematics skills. (2) The application of core 
concepts and principles from all subject matter 
disciplines. (3) Becoming a self-sufficient in- 
dividual. (4) Becoming a productive member of a 
family. (5) Thinking and problem solving. (6) Con- 
necting and integrating knowledge. 

The Kentucky Education Reform Act represents 
an integrated model of school reform. All of the 
major parts relate to each other in significant ways. 
For example, funding issues are tied to school 
performance. School performance is related to 
school-based decision-n:iaking, and so on. It ap- 



pears that the performance-based assessment design 
will have implications for the kind and amount of 
information available to implement other parts of 
KERA. It is for this reason that it is critical that 
the performance assessment component is designed 
and executed well. 
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Colleen Lyonj 


Leslie 


Ernest C. Taylor 


Marion 


JoAnn Fortney 


Laurel 


Steve Hutton 


Beechwood, Ind. 


John Chadwell 


Middlesboro Ind. 


Carl Hicks 


Beechwood Ind. 


Paul Spangler 


Middlesboro Ind. 


Brenda Anderson 


Nicholas-Carlisle 


Edwina C. Gibson 


Monticello Ind. 


Conrad E. Young 


i^iaric 


Delores Sapp 


Montgomery 


Joseph R. Jones 


i^iarK 


Barbara Amburgey 


Montgomery 


Marge Brackett 


Clark 


Sharon Noble 


Hazard Ind. C 


Jim Welb 




Paiii Price 


Robertson 


Ray Snowden 


Estill 


Doris M. Powell 


Todd 


Tom Hunt 


Mo ntgomery 


Conley Manning 


Williamsburg Ind. 


Paul R. Spangler 


Middlesboro Ina. 


Jeanne Meacham 


Robertson 


Patsy R. Lester 


Washington 


Frank Howara 


Fayette 


Margaret Morriscy 


Kenton 


Mat Young 


Lawrence 


Donnie Collins 


Hazard Ind. 


John O'Brien 


Ft. Thomas Ind. 


Darnel L. onuU 


Beechwood Ind. 


Linda France 


Jessamine 


Kicnard r. oowung 


Leslie 


Debbie Egan 


Jessamine 


Jeniuter Hubbard 


Kenton 


Sandra Bailey 


Knott 


David S. Kandolpn 


Boyle 


Connie Aubrey 


Jessamine 


Jerry L. Leber 


Boyle 


James C. Hanks 


Marion 


Jack Tucker 


Bourbon 


David Hagan 


Marion 


Kjrby M. Wngnt 


Mason 


Diane H. Evans 


Marion 


Jane H. (..iine 


Mason 


Ruby M. Wright 


Mason 


Yvonne Slusher 


neil 


Olen Taylor Collins 


Leslie 


Phyllis Whitaker 


Magoffin 


Anne Keene 


PtKeville Ind. 


Guy Strong 


(^laric 


Mitchell Thompson 


Marshall 


William ooiton 


(^laric 


Ken Rosenbaum 


Jefferson 


rai rresienocrg 


IV I vaiicy cu. ^uupcrauvc 


Barbara Helm 


Jessamine 


wanoa c vice 


vauey xiu. ^^oope^a^vc 


Ivan Cooper 


Kenton 


Jane Komara 


Fayette 


Sue G . Magee 


Jessamine 


rat Nickell 


Fayette 


James W. Burks 


Bourbon 


I ruiena i ate 


CSlIll 


James Whitaker 


Jefferson 


Carolyn Clark 


Fayette 


Barbara Stenzel 


Jessamine 


Janice Led ford 


Asniand ina. 


Yvette Blair 


Knott 


Olenn A. Kieoei 


Ashland Ind. 


Melissa Martin 


Knotl 


oayie oamaro 


Elizabeth town Ind. 


Wendy Lakes 


Jessamine 


Kennetli Barnard 


c4LcaDcuiiown inu. 


Mike Norris 


Jefferson 


Brenda Priddy 


Elizabethtown Ind. 


Tom Mills 


Boyle 


Katherine Schloemer 


Louisville Parochial Schs Jefferson 


Michael H. Byers 


Hardin 


James L. Thomas 


Estill 


Ed Murphy 


Fayette 


Dale Brown 


Carroll 


Richard A. Steben 


Fleming 


Ed Watkina 


Fayette 


John Farler 


Hazard 


Ron Pelfrcy 


Fayetle 


Glen A. Hendrix 


Leslie 


Patricia Miller 


Estill High 


Karen S. Pravler 


Mercer 


Joseph Clark 


Fayetle 


Johnnie Slonc 


Mercer 


Bill Burch 


Ashland Ind. 


Paul K. Goodin 


Pineville Ind. 


Herb ConJey 


Ashland 


Thonus Sizemore 


Leslie 


Loii Pritchen 


Fayette 


Paulette Ofliitl Kutter 


Mercer 


Betty Beardsley 


Fayetle 


LaDonne Walker 


MarshaU 


Jim Schlich 


Fayette 
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Conference Participants 



Name 


County 


Wilbum Nons 


Leslie 


Frank W. Baker 


Leslie 


Calvin Crawford 


Leslie 


Damon Huff 


Leslie 


Offina Shepherd 


Leslie 


Janie Ohr 


Estill 


Clayton Johnson 


Estill 


Patti Abney 


Estill 


Don F. Trivette 


Fayette 


VikJd Clemens 


Campbell 


Roger Meek 


Boyle 


Jay Carroll Miller 


Hardin 


Carolyn Murphy 


Fayette 


Ina Bradford 


Carter 


Jo Ashworth 


Carter 


Anthony Johnson 


Augusta Ind. 


Linda Roberts 


Henry 


Paula Martin 


Henry 


Tim Abrams 


Henry 


Jo Mitchell 


Henry 


Sharon Silvers 


Henry 


Nancy Sutton 


Hazard Ind. 


Jennifer Green 


Jefferson 


Carole Ramey 


Jefferson 


Martha Sandusky 


Marion 


R. B. Benningfield 


Marion 


Robert Davenport 


Marion 


Judy Gaddie 


Marion 


Jane Becker 


Nicholas 


Alfred M. Collins 


Perry 


Yvon Allen 


Perry 


Charles R. Campbell 


Perry 


Dennis P. Wooton 


Perry 


Bo McCreary 


Model Lab School/EKU 


Batbara Disney 


Paris Middle 


Patti Price 


Robertson 


Dorothy Lester 


Russell Ind. 


David M. Heath 


Perry 


Lois Hatfield 


Pulaski 


Michael Bromagen 


Bath 


Dulcie Hardin 


Crittenden 


Nancy Sulion 


Crittenden 


Susanne Hoagland 


Madison 


Donna Craft 


Madison 


Michael White 


Pineville Ind. 


Hazel S. Powell 


Middlesboro Ind. 


Sue W. Roberts 


Marshall 


Shelly Cohen 


Jefferson 


Teresa Petot 


Paintsville Ind. 


Bruce Bonar 


Model Ub School 


Jacqueline G . Vance 


Model Lab School 


Leanna Austin 


Mayfield Ind. 


Hsie Jones 


Mayfield Ind. 


Bonita Lykina 


Mayfield 


Carole Hancock 


Pulaski 


Reece Little 


Jefferson 



Name 


County 


Howard Hardin 


Jefferson 


Sandra Poe 


Jefferson 


Nan Franck 


Jefferson 


Sandra Lyvek 


Scott 


James Stewart McPcek 


Pike 


Roy Chapman 


Woodford 


James Clay 


Pike 


Cheryl Cox 


Powell 


Susan Miller 


Powell 


Martha Staker 


Scott 


Shelby Reynolds 


Rockcastle 


Twyla Hanna 


Hardin 


Sandy Greenwood 


Hardin 



